Methods for acquiring ultrasonic data

ABSTRACT

Methods for acquiring ultrasonic data are disclosed. An image-acquiring system is provided. A three-dimensional target region is selected. A plurality of fiducial positions corresponding to anatomical features in the target region are calculated. A model of the target region comprising a plurality of target locations representing a plurality of planned locations in the target region at which ultrasonic data is to be acquired is created, and a visual representation of the model comprising a plurality of graphical elements is displayed. Ultrasonic data at each of the planned locations is acquired. A transformation of the visual representation is executed, comprising: performing a data quality test at each target location; for any target location that fails the data quality test, altering a graphical element corresponding to the failed target location to indicate failure of the data quality test at that location; and displaying a transformed visual representation comprising updated graphical elements on the visual display.

RELATED APPLICATION DATA

This application is a continuation of U.S. patent application Ser. No.14/568,138, filed Dec. 14, 2014, which claims priority from U.S.Provisional Patent Application Ser. No. 62/040,007, filed 21 Aug. 2014,and U.S. Provisional Patent Application Ser. No. 61/918,664, filed 20Dec. 2013. All of the forgoing are incorporated by reference herein intheir entireties.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to the field of diagnostic non-invasiveimaging and particularly to ultrasound imaging.

2. Background of the Art

Ultrasound imaging is a well established procedure in a wide variety ofconditions, including its use for examination of fetal and maternalhealth during pregnancy. There are established procedures for trainingpersonnel, both in the acquisition of images with hand-held ultrasoundtransmitters, and in analyzing them for diagnostic as well as metricpurposes. Standard metrics such as fetal dimensions and derivedparameters help quantify fetal health and other diagnostic issues. Inaddition, there is continued effort to develop computer code that canautomatically analyze an ultrasound image scan to compute these metrics,though the use of such algorithms is not yet widespread. Such code,while it automates the determination of the metrics from the images,does not in any way diminish the need for skilled operator acquisitionof the images themselves.

The introduction of high quality ultrasound scanning equipment that canbe attached to personal devices such as smartphones and tablets raisesthe potential for widespread dissemination of scanning technology, inparticular to remote and rural areas. Developing countries includingthose in the Indian subcontinent, Africa, and elsewhere could becomemajor beneficiaries of the technology. However, two factors inparticular militate against this expansion in use and benefit: trainingneeds, and (at least in India) heavy regulation against thepossibilities for misuse. We here describe and add to the systemreferenced above, in the context of these factors and the way the systemenables wider use of ultrasound in training, screening and the guidanceof therapy. Its use thus has the material consequence that many morepatients (particularly in a rural setting) may be referred fortreatment, and may be treated more accurately, than might occur withcurrent technology.

It is desirable to be able to provide an ultrasound-using health systemfor significant investment in the training of personnel to acquiresuitable scans, which can then be read by experts to arrive atconclusions. (While it is possible for an expert to acquire a scandirectly, the expert's time is a limited and costly resource, best notused in acquisition, where travel and other capital and personnel costsare required.) Particularly in developing countries, there is a lack ofwell trained personnel in the field. A newer generation of ultrasoundmachines has made the skill of acquiring good ultrasound images morereadily teachable to people without prior skills. However, these imagingprocedures depend on the operator's training in obtaining images, whatto look for in the images, and ensuring that the result is a scaninterpretable by skilled diagnosticians. While the learning curve isbecoming shorter, imaging results remain highly dependent on traininghuman visual interpretation of the images displayed from an ultrasoundscan.

The Necessity of Anatomical Guidance in Current Practice

The present invention aims, not to improve anatomically orientedtraining, but to completely remove the necessity for the trainee tolearn internal anatomy and its use in guiding image acquisition.However, as background, we discuss in further detail why the existingpractice demands anatomical knowledge, and hence acts as a trainingbottleneck, with some further consequences considered below. Somehardware elements of existing practice are adapted for use in thepresent invention, making their availability important, so we givespecific examples here.

FIG. 1 shows an exemplary instance (discussed in more detail below) ofhow training is commonly performed within the scope of current practice.Under the supervision of a trainer 104, the trainee 103 operates ascanner 102 on the body, or part of the body, 101 of a patient. Thisscanner or probe comes in a variety of embodiments from linear probes tocurved linear arrays, phased arrays and volume probes. We are concernedhere primarily with the so-called B-mode (brightness mode) or 2D mode ofultrasound, in which a linear array of transducers simultaneously scansa plane through the body that can be viewed as a two-dimensional imageon screen, but other modalities (such as three-dimensional ultrasoundimaging, for which software and apparatus are commercially available)exist in current use and may be adapted within the spirit of the presentinvention. Two-dimensional, hand operated ultrasound imaging has a longhistory, with key events such as the 1965 release of the SiemensVidoson™ scanner, the first commercial B-mode scanner that operated inreal time, producing a current planar display which could be used toguide a change of scanner position and hence of the acquired image. Atthe high end, a General Electric (GE) system which is fairly typical ofthe features available for this class of machines is the LOGIQ™ series,such as #7 and its associated transducers and probes. The use mode ofguidance by watching the currently acquired image remains typical, andthe present invention seeks to replace it. Although the invention doesnot seek to modify the functioning of the scanner itself, the processseeks to educate and train a local user in the art of appropriatelychanging the scanner positions used locally. In present practice thelocal operation of scanning equipment is managed by observing the imagesacquired. (In the present invention it is managed with the help of atracker reporting location and position of the hand-operated components,and by other additions described below.)

Currently used in the same anatomy-guided manner are volume transducers,that, held in a single position, can capture data from material pointsnot confined to a single plane, and thus generate a 3D image withoutfurther reconstruction. More complex and expensive devices can modifythe plane somewhat without moving the scanner, or acquire ‘volume’ datafrom multiple planes simultaneously, such as in U.S. Pat. No. 7,758,509B2, Multiple scan-plane ultrasound imaging of objects, Angelsen andJohansen, or see the review “Three-dimensional ultrasound imaging” byFenster, Downey and Cardinal, Phys. Med. Biol. 46 (2001) R67-R99(www.iop.org/Journals/pb PIE S0031-9155(01)12089-0) which remarks that“Although the integrated 3D probes are larger and heavier thanconventional probes, they are usually easy for the operator to use.However, they require the purchase of a special ultrasound machine thatcan interface with them.” A representative example has the productnumber 4D3C-L from GE. However, while this eases the task of anatomicalguidance (since it is easier to recognize structures in a 3D image thanin a slice), such local and hand-manipulated guidance is still reliedupon in current medical usage. The requirements that the user knowanatomy, and that the user sees the anatomical images, remain in placefor actual diagnostic practices including technician acquisition of theimage data. In the description below of the present invention, wedisclose how these requirements may be avoided.

Also available from GE is a scanning system and probe that is morelikely to fit within the scope of application of this invention indeveloping countries. This is the handheld VScan™ pocket ultrasound.Other manufacturers have competing products with different names andmodel numbers, and these could equally well serve as examples ofcommercially available system for use in the present technology.

In training or in regular use, the scanner 102 is normally placed incontact with the abdomen (or may be inserted into a body cavity) orother area where diagnostic or training imaging is to be performed, withsufficient coupling for good transmission of ultrasound, and collectsdata from echoes in a region 112 which has a shape and spatial relationto the scanner 102 which is fixed by the scanner design. As trainee 103displaces or rotate the scanner 102, along or about any axis, the region112 moves and turns correspondingly. (Most commonly the region 112 is aplanar ‘partial fan’ as shown, though the more sophisticated scannersmay without a change in device position acquire data from echoes inmultiple planes, thus forming a directly volumetric scan, as mentioned.)The data obtained from the region 112 are fed via a wireless link orwire 111 to a computer 113, here shown as outside the casing of thescanner 102, though in general this need not be the case. Wireless ismaking rapid advances at the present time, and even particular protocolsare quickly increasing their bandwidth capabilities. Furthermore, theconcept of software-defined radio, exemplified in the open source GNUradio (allowing kilobits per second transmission) or Microsoft's SORA™system supporting many megabits per second allow software andalgorithmic advances to be immediately incorporated in contrast to moreconventional application-specific integrated circuit (ASIC) or fieldprogrammable gated arrays (FPGA) approaches. It is sufficient for thepurposes of this invention that conventional mobile telephone links beavailable, but the link to the computer from the scanning device caneasily follow Wi-Fi (or more technically the IEEE 802.11 protocol)allowing much higher bandwidths. The computer 113 converts theultrasound echo data to an image (by commercially available softwareprovided by the ultrasound manufacturers with their systems) which isdisplayed conventionally via a link to a display unit (also referred toherein as monitor) 105 which may or may not be integrated or attachedwith a hand-held scanner 102 and/or the computer 113 which converts echodata usually to a planar image 106, displayed on a monitor or similardevice 105. We do not discuss in detail the display units used incurrent practice, since first, the local display of the image data ispreferably avoided in the present invention (we describe in more detailbelow the display that the present invention does use), and second, theunits are just conventional screens available on smartphones, laptops,and the like and can range in size from the handheld such as the GEVScan™ pocket ultrasound to ‘desktop’ units like the GE LOGIQ 7™, bothalready mentioned. The processing of echo data to form images isconventional and includes both analog and digital processing steps, suchas described for example in Chapter 1 of the open source book“Ultrasound Imaging” edited by Masayuki Tanabe and published by Intech,2011: http://www.ti.com/lit/an/sprab32/sprab32.pdf online) or in fastgeneral purpose processors.

Both the trainee 103 and the expert trainer 104 can view the planarimage 106 (if the trainee is given authorization to view the image),shown as a sector scan in the fixed plane of the display unit 105,rarely in the same direction of gaze as the scanner 102 for eithertrainee 103 or expert 104, and almost never in the moving orientation ofthe acquisition region of the body 101. (This mismatch, also typicalwith the more advanced volume scans, greatly increases the cognitivedifficulty of their tasks.) The expert then may (verbally or by text toa local view screen) instruct the trainee to look for characteristiclandmarks in the planar image, to arrange that a single planar image 106should suffice for a particular clinical purpose. For example if theangle of, and the pressure applied to, the scanner is not ideal relativeto the disposition of the fetus, then characteristic features used inmetrics that indicate fetal health, may not be visible. The expertsuggests alteration of the angles and the positioning of the probe ingeneral, so that a useful scan maybe obtained. Similarly, judgment ofthe anatomical position of the planar image 106 is needed to move thescanner 102 so that a single planar image 106 clearly reveals anappropriate slice of the femur (as an example of a diagnostic ortraining target, shown in FIG. 2 and discussed further below). Softwaremay be used to provide automatic corrections based on automated imageanalysis for orientation and operation of the distal sensing devices,but as long as the goal remains optimality for one or more discreteslices, it is hard to operate without human judgment, and individualskill transfer of such judgment from expert to trainee, becauseanatomical clarity plays a large role in defining such optimality.

FIG. 3 shows the above-described current training process as a moreabstract workflow. The scanner 302 interacts 308 with the patient ormodel 312, using echoes to obtain data related to material in itscollection region 301. These data are processed to a greater or lesserdegree, and communicated 311 to the computer 313 which continues theprocessing, resulting in a digital image which is passed 310 to themonitor or similar device 305 which displays it. This is observed 315 bythe trainee 303 and 317 by the expert 304, who gives 316 spoken or textadvice to the trainee 303, who modifies 323 the location and orientationof the scanner 302 accordingly, and the workflow continues on the samepaths. The Figure does not include the trainee's mental models forpatient anatomy, quality of images, etc., or the learning process bywhich these are created and improved.

In the existing practice this process of training is continual,typically continuing beyond the specific training and licensing periodinto actual practice. The trainee or neophyte operator acquires therequisite skills by a learning process that is largely based on anexpert (local or remote) distinguishing ‘good’ scans from ‘bad’ ones andrelaying approval and corrections to the trainee. We illustrate thiswith an example, to elaborate on the point that the distinction is notmerely technical (with such criteria as resolution and level ofdistortion, expressible in engineering terms) but anatomical, requiringreal anatomical expertise on the part of the operator: even if theoperator is not responsible for diagnostic or clinical decisions, oradvice to the patient, this expertise is required for the acquisition ofclinically useful slice images. Our example is the particular fetalparameter of the length from end to end of the femur (the thigh bone,longest in the body): similar considerations apply to other fetalparameters, and to many non-obstetric applications of ultrasound. Toacquire a slice image that will be useful in estimating femur length,the user must understand the anatomy of that moderately complex bone(FIG. 2). The parameter is defined as the distance along the femur shaft201, and should not include the secondary bone forming center at themore complex end. To assess this by means of a planar image requiresthat the image plane intersects the bone similarly to the plane 221(shown sideways as a line, as if from the scanner head) or to onerotating the plane 221 about the poorly defined axis of the curvedshaft. The femur of a fetus may present in any orientation, depending onfetal posture, and many planes are impossible for scanning from pointson the abdominal surface, so the operator must select a full lengthwiseslice from what is achievable and do it using only what is visiblewithin the slice. The knob 250 to one side is not visible when the sliceis correct, and a slightly oblique slice can miss one end or the other.Even a technician who is not tasked with the measurement itself mustunderstand the femur length parameter well to produce images from whichit can reliably be estimated.

It is equally problematic to estimate the length of the humerus (asmaller and more delicate bone though geometrically slightly simpler),and parameters such as head circumference, occipitofrontal diameter andabdominal circumference are even more difficult. Only anatomicalunderstanding, and substantial experience of what an acceptable slicelooks like, can produce a high proportion of good images: similarly fordiagnostic images of the liver, gallbladder (such as gallstones),pancreas, thyroid gland, lymph nodes, ovaries, testes, kidneys, bladderand breast. For example, it can help to determine if an abnormal lump inone of these organs is a solid tumor or a fluid-filled cyst, or detectabnormal widening of blood vessels. A newly qualified technician in thefield creates more ‘bad’ images than an experienced one, reducing theefficiency of the system, but only on-the-job continued learning, withcontinued feedback from experts, produces the final level of skill. (Thenon-quantitative distinction between male and female fetus is moreeasily learned and practiced.)

The present invention does not aim to provide the operator with feedbackon the quality of planar slices, in the above sense of diagnosticquality: to do so would require replacing the expert's anatomicaljudgment with artificial geometrical intelligence at a higher level thanhas yet been shown achievable, and would at most improve training in theclassic, anatomy-guided process. The aim is to completely replace theanatomical criteria on acquired slices, reducing the cognitive task, therequired training, and the educational level required of a trainee, allof which are bottlenecks in the dissemination of ultrasound diagnosis,particularly in less developed countries. At the same time, it avoids amajor problem with current practice, described next.

The Informational Consequences of Local Access to Images

We have seen above that visual feedback is necessary in the currentsystem, if the operator is to acquire good images. Since that feedbackinvolves anatomical knowledge, it creates a bottleneck of availableultrasound operators and a difficulty in training more, but there isalso a problem with the fact that such feedback requires that the devicebe operated by somebody who from moment to moment, sees its output as ananatomical image. This causes a significant obstacle to dissemination ofthe new generation of highly portable cheap scanners. We referparticularly to India, but the problem is widespread. There are groupswho use ultrasound imaging to determine sex, as a means to selectivelyabort (in particular) female fetuses. India's 2001 census revealed aratio of 126.1 male births to 100 female in the Punjab, showing thatmany parents had practiced selective abortion: but statistics give noclue as to which parents did so. The birth of a son, or several sons, inone family is not evidence of an aborted daughter. To limit this use ofultrasound technology, India has laws that disbar physicians or medicalpersonnel who inform parents of the sex of the fetus. However, if alocal operator or clinical worker can see the images of a fetus, thatperson can be bribed to reveal the sex. To attack that problem, theIndian government has also severely restricted the deployment ofultrasonic imaging equipment for any purpose whatever, particularly inremote areas, outside the larger and more controllable clinics. Inseveral states, the sale of portable ultrasound is a criminal offence,because it can abet in the crime of sex determination. Theserestrictions render the technology unavailable for ethical diagnosticpurposes, even those unrelated to pregnancy, and prevent both businessesand non-profit organizations from supplying ultrasound services in manyareas of India and other countries.

If the rural operator could acquire clinically useful data which is thentransferred to a controlled clinic, using equipment that does not enablethe operator to access images that reveal fetal sex, such restrictionswould be unnecessary. However, since the present system requires thatthe operator sees the current image (in order to use anatomicalawareness to improve it), it cannot simultaneously blind the operator tofetal sex.

A similar problem arises with the privacy of such images: while mostcountries have privacy laws restricting the availability of patientinformation (the Health Insurance Portability and Accountability Act of1996 in the US being a notable example), the more people see a medicaldatum, the greater the risk that it is shared improperly. This isparticularly true in small communities, where a patient may bepersonally known to an operator.

Thus there is a need to develop a method that ensures widespreaddissemination of the technology and its use for the benefit of people,and at the same time, avoids use of the technology for unethicalpurposes. The invention described in the above disclosure, and itsextensions here disclosed, avoid the use of anatomical awareness innavigation. This not only simplifies the task (as noted above) butenables the acquisition of image data while maintaining their securityagainst local knowledge, and hence of improper use of such knowledge.This has evident medical, social and business advantages.

Three-Dimensional Reconstruction in Current Practice

If the instantaneous position of the scanner is tracked by an attachedtracking device, parameters obtainable from the scanner specificationsallow computation of the planar extent and position of a scan from aspecific location of the scanner. The volumetric scan required is thenobtainable from a succession of such planar scans, without requiringthat these be parallel or move by uniform translation, which would bemore easily achieved by a robotic system than by a human. There are manymeans of achieving a volumetric data set from such data. Inillustration, we observe that every intensity level at every pixel ineach image can be ‘splatted’ to an (x,y,z) position in the displaycoordinate space, analogous to the splatting of ray values into a 2Dimage described in Westover, SPLATTING: A Parallel, Feed-Forward VolumeRendering Algorithm, doctoral dissertation, UNC Chapel Hill 1992.Combining these splats, by blending and interpolation, one can constructan array of intensity levels I(i,j,k) which can then be manipulatedusing standard techniques for 3D scans by CT, MRI, etc., which areengineered to produce data in this format. See for example the PhDthesis 3D Freehand Ultrasound: Reconstruction and Spatial Compounding byR N Rohling; but it is typical that for instance the paper Freehand™ 3DUltrasound Calibration: A Review, by Hsu, Prager, Gee and Treece,CUED/F-INFENG/TR 584, December 2007, which discusses accuracy in somedetail, does not consider guidance of the user. A web site reviewingscanning products,(http://kpiultrasound.com/4D-ultrasound-machines/View-all-products.html,retrieved 12 Dec. 2013) remarks that Freehand™ 3D “requires operatorskill and hand movement to do the 3D scan”. Such skill is hard toacquire if the operators' use of the scanner is guided by the anatomicalimages: the present invention also uses three-dimensional reconstructionfrom a series of planar scans, but provides guidance whose use iseasier, and easier to learn. This has the material effect of shorteningboth use and training, increasing the pool of potential trainees byreducing educational prerequisites, and (with appropriate funding)increasing the actual number of qualified operators beyond what ispractical with current methods.

In the system disclosure referenced above, an operator manipulates anultrasound scanner, guided not by a display of the image it is currentlycollecting but by a geometric view of where the image is beingcollected, as in the shaded polygon 555 in FIG. 5. This polygon 555moves in geometric correspondence to the acquisition region of thescanner, with the requirement of sweeping through an indicated volume ofinterest such as 505. It is not of the essence of the invention thatonly geometry be displayed, though this is an important option inconcealing from a local operator the imaging details acquired, to avoidrevealing fetal sex. The display of 3-dimensional geometric guidance inmotion of the sensor is central to the invention: the un-usability ofthe system for sex determination, in certain implementations, is one ofthe more important options that it makes possible.

USPTO Patent Application 20090169074, System and method for computerassisted analysis of medical image addresses the sex revelation problemby selective blurring. The performance of such selection requirescomplex real-time image analysis connected to the device, identifying byone means or another (many are listed as possible) the anatomical regionto be masked. It is not trivial in a two-dimensional anatomical slice toidentify what is on view (even for a human observer), and to the best ofour knowledge the particular task of recognizing fetal genitalia in realtime in such scans has not been accomplished. This task is not calledfor, either in our application referenced above, or in the extensionhere disclosed.

We note also the literature on 3D reconstruction of a scanned tissuebased on the tracked motion of a sensor: see for example “AnInertial-Optical Tracking System for Portable, Quantitative, 3DUltrasound” by A. M. Goldsmith and P. C. Pedersen, 2008 IEEEInternational Ultrasonics Symposium Proceedings, pp 45-49, andreferences therein. We do not here claim innovation in the algorithms bywhich the reconstruction is performed, but in the method by which theoperator of the sensor is guided. The paper “Interactive Training Systemfor Medical Ultrasound” by C J Banker and P C Pedersen illustrates wellthe current method of guidance, by visual anatomical feedback imagingthe interior of a real or virtual patient: the twin disadvantages ofthis are the considerable training required, and the (less demanding)utility for fetal sex determination, which in some situations isillegal. The present invention offers more easily mastered geometricalguidance, which by not displaying the patient's internal anatomysimplifies both the task and compliance with local legal requirements.

As further background, we here recapitulate a summary of the abovedisclosure, before reciting a brief description of the present extensionto it.

SUMMARY OF THE INVENTION

We here describe various additions and possible modifications to theinvention previously disclosed and described above, with particularemphasis on the manner of local display and transfer of data.

In a first such variant, the currently acquired image is displayed insitu in the geometric view shown in FIG. 5, as a ‘texture’ added to themoving polygon 550, rather than as a flat view in an unchanging positionon a display device. If this view is so oblique as to obscuresignificant details, that the operator wished to see in real time, a‘laid flat’ display unit 105 as in current practice may be added.

In a second such variant, the currently acquired image is displayed inthe same real-time way as a texture on the polygon 150, in the sameapparent location, but sufficiently blurred (throughout) that detail ofthe scale of fetal genitalia can nowhere be seen.

In a third such variant, geometric features including but not limited toboundaries between light and dark areas are extracted from the currentimage data by means well known to those skilled in the art, without anattempt to attach to them an anatomical meaning (such as ‘edge of afetus leg region’). If such an extracted boundary is long and smooth,with few corners or sharply curved regions, it cannot represent thesurface of fetal genitalia, and may thus be shown in a polygon 150 or210, or both, without revealing fetal sex but assisting the operator inanatomical judgment of the scanning location.

In a fourth such variant, a preliminary analysis of the echo datalocates points whose sound reflections are sufficiently strong, relativeto the other echoes currently received, to imply that these points areoccupied by bone or other hard material. These points are added to thedisplay.

In a fifth such variant, features such as those described in the thirdor fourth are displayed in superposition on the blurred image shown asdescribed in the second variant above.

In a sixth such variant, numerical measures are applied to the acquiredimage data, revealing whether they are suitably sharp. If they are not,the region within the target box of the display unit 105 where betterdata are needed can be three-dimensionally marked (by a smaller box, orother graphical means familiar to those skilled in the art) as requiringa repeat visit by the region 112 acquired by the scanner.

In a seventh such variant, numerical measures are applied to theacquired image data, revealing whether their planes are sufficientlyclosely spaced for the reconstruction of a volume data set. If they arenot, the region within the target region of interest 505 where betterdata are needed can be three-dimensionally marked (by a smaller polygon555, or other graphical means familiar to those skilled in the art) asrequiring a repeat visit to the region 112 by the scanner.

In an eighth such variant, the sensing system is capable of Doppleranalysis, for example giving indications of maternal and fetal bloodflow along the line of ‘view’ by frequency changes in the echo. Sinceother directions are often important, data collection requires varyingthe direction as well as the point with respect to which data arecollected, in a manner described in more detail below, giving a fullerreconstruction of flow than is possible with image acquisition in asingle plane. The system applies numerical tests for the confidencevalues of this reconstruction, and displays the results in a polygon 550or 810, or both, to guide rescanning.

A ninth such variant allows for movement of the target during the scan,such as a fetus may do, by fitting the motion of bones to an overallmodel of motion, and using this to correct for artifacts of movement.

A tenth such variant allows for more regular and limited motion, such asthat of the heart. In this case the motion is unavoidable, but periodicrather than unpredictable. Collecting the scan data together with timestamps and the output of at least one electrocardiogram or otherpulse-related sensor, and assuming regularity of the motion, we collectecho data for a sufficient density of points (x,y,z,Φ), where (x,y,z)represents spatial position and Φ is cardiac phase. The acquired imageis thus four-dimensional (giving either scalar echo strengths or vectorflow data), and the repetitive nature of heart motion means that gapscan be revisited.

In an eleventh such variant, the three-dimensional image reconstructedis used for the automatic analysis of anatomical structures such as agrowing femur, including quantitative comparison with growth norms,objective sex identification to be non-erasably stored (permittingcorrelation of a clinical institution's records with later abortions, toraise statistical warning of security leaks), identification ofstructurally important features, and for certain conditions (such asidentification of conjoined twins, mis-positioning of a fetus within thewomb, or inappropriate arterial connections in an adult or immaturebody) identifying plans of treatment.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a representation of conventional training of personnel and theuse of apparatus in the acquisition of scanned ultrasound images forhealth monitoring including fetal health monitoring.

FIG. 2 shows a femur and an imaging slice orientation which permitsestimating its length.

FIG. 3 is a schematic showing the information flow associated withconventional training as in FIG. 1.

FIG. 4 is a schematic representing the information flow associated withtraining in the use of the present invention.

FIG. 5 schematically shows a geometric relation of an image-acquisitionplane in a patient-fixed frame of reference, as shown to a fieldoperator as a geometric form in a 3D display.

FIG. 6 schematically shows information flow in deployed use of thepresent invention, including connection with a separated clinicaldisplay, to which anatomical visualization may be reserved.

FIG. 7 is a flow diagram showing data flow in a process according to thepresent technology, and certain stages along the data flow where accessby the local user may be enabled or prevented.

FIG. 8 schematically shows the relation between a pattern in theacquisition region of a sensor, and the current way to display thatpattern.

FIG. 9 schematically shows patterns in the acquisition region of asensor, and blurred versions of those patterns.

FIG. 10 schematically shows edge curves extracted from the patterns inFIG. 9, simplifications of those curves, and their superposition on theblurred versions in FIG. 10.

FIG. 11 shows the work flow of image acquisition according to thepresent invention, for a static target.

FIG. 12 shows the work flow of image acquisition according to thepresent invention, for an irregularly moving target such as a fetus.

FIG. 13 shows the work flow of image acquisition according to thepresent invention, for an periodically moving target such as a heart.

FIG. 14 shows the overall work flow of acquiring, using and reporting ofultrasound imagery, according to the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Our invention is directed towards both the problems above, of trainingneeds, and legal restrictions against the possibilities for misuse: theneed for skill in acquiring satisfactory images by visual inspection andanatomical evaluation, and the consequent necessity of visual access bythe operator, with attendant risks of selective feticide and loss ofprivacy. By blinding the local user to the images, distal qualifiedmedical personnel may retain control over diagnoses and the images forbusiness and professional purposes as well as legal and billingrequirements.

This technology enables a method, system and apparatus to de-skill therequirements for the ‘field operator’ who manipulates the ultrasoundequipment to acquire the images, reducing both the prerequisite level ofeducation, and the time and cost of training. Further, implementationsof this method can be easily adapted to prevent unauthorized people(including the local field operator) from viewing, sharing or analyzingthe images.

The training of field operators requires no more technology than thatwithin a simple 3D video game. The training may be provided in arelatively brief period, with appropriate feed-back through processesand systems described herein. Both the use of the method and apparatus,and the necessary training, can thus cost less than for current methods,and importantly can be widely deployed in remote areas. (Wired orwireless web access is a necessary condition for deployment of theanalysis component, although high bandwidth and lack of interruption arenot critical to the performance of the method.) Furthermore, in contrastto existing training methods, the present invention for training doesnot require the acquisition or display of any patient images to thefield operator, because when working with a patient the field operatorwill see only computer-generated 3D geometric constructs. As notedabove, the absence of locally viewable actual images of a patient is oneoptionally desired objective of the present technology, especially injurisdictions where such local sharing may be prohibited by law. Inconsequence, in its deployed version, the technology can simply omit thesystem elements required for local display, or a further layer oftechnology can actively prevent (for example by encryption) display ofthe images to the field operator. The technology does not exclude thatthe system be physically equipped for local viewing, in which case itmay be provided with a distal command that can deny or block local imagecapability: however, this capability raises the hardware costs of thesystem, and offers a weak point to security in that an engineer who candisrupt such blocking (by modifying communication with the distalcomputer, by local interference with the command, or otherwise) thenopens the entire local system to display of the acquired images, withouthardware changes. Where security is important, therefore, in ourpreferred embodiment the local, on-site system excludes entirely thecapability to display image data captured from the ultrasound sensor. Itis necessary, as described below, to have a visual display system thatcan show (with one or more depth cues to a human viewer) athree-dimensional geometric outline of the volume for which data are tobe acquired, and a flat polygonal structure whose position representsthe three-dimensional position of the planar region for which the sensor(as currently located) can acquire images. It is not necessary todisplay any anatomical information, either simulated or acquired fromthe current patient, in direct form or enhanced as in A Virtual RealityPatient and Environments for Image Guided Diagnosis, by Takacs, Hanakand Voshburg, Medical Imaging and Augmented Reality, 279-288: theconcern there is to show the anatomy more clearly, whereas the objecthere is to avoid all need to show it. In our preferred embodiment, nosuch information is shown.

This lack of need for anatomical display in guidance enables confiningsuch displays to authorized centers for diagnostics, while the geometricdisplay described below provides both training to potential local usersand guidance to actual local users, without requiring that the actualuser sees any anatomy or (hence) that the trainee needs to see anyanatomy, real or simulated. Technologically and administratively secureprocedures can then safeguard reservation of the use of such imaging toethical and beneficial purposes. Furthermore, the framework of dataacquisition concurrently with 6-degree-of-freedom data for theultrasound device makes possible the reconstruction from individualimages (in our preferred implementation, planar images) of a3-dimensional model of the scanned anatomy, if the acquired planarimages collectively satisfy appropriate geometric criteria (rather thanindividually satisfy anatomical criteria), and hence the quantificationof diagnostic measures from such a 3-dimensional model. Unlike thecurrent anatomical criteria, the geometric criteria can bestraightforwardly quantified using only tracking information, so that acomputer can provide feedback to a user or trainee without analysis ofthe acquired images, as to whether the geometric criteria have been met.This has advantages in simplifying and improving training and use, evenwhere security is not a concern. FIG. 4 shows the flow of the simplifiedtraining process enabled by the current invention.

In one preferred embodiment, the system uses only a moving scanner thatcan acquire a planar image, in a fixed position relative to the scanner,rather than the volumetric scanners mentioned above. These more costlysystems could be used, and their output could be locally hidden in thesame way, with minor modifications of the reconstruction and displayperformed distally, as will be apparent to those skilled in the art.However, since the present invention includes 3-dimensionalreconstruction and distal 3-dimensional display, the relative advantageof such systems is less than in it is in anatomy-guided practice. (It istrue that to capture the motion of a structure such as the heart, asequence of volumes has greater value than a sequence of slices. Withinthe spirit of the present invention one could use such a sequence ofvolumes, acquired by geometrical rather than anatomical guidance, toconstruct a 3-dimensional movie.)

The use for scanning in pregnancy is a preferred application of theinvention, which exploits both the reduction in required field skill andthe option of restricting display, and we therefore use this forexemplary description. However, the extension to other ultrasounddiagnostics (e.g., orthopedics, gastroenterology, and other ultrasoundimage-enhanced views of a patient), where a lack of skilled operatorsand the possibility of abuse are currently preventing a neededwidespread dissemination of the technology, should be obvious to thoseskilled in the art.

The communication bandwidth requirements for the system are relativelymodest. An uplink 670 (FIG. 6) to the cloud 680, and thence delivery 685to the clinical display site (hereafter also denoted site) 690 where theimages are to be read (or in a less preferred embodiment, a direct linkfrom the computer 613 to the clinical display site 690) does not have tobe 4G, since the images can be sent over a period of time provided theycan be stored within the handheld device. The downlink requires evenless bandwidth, since only a report of the diagnostic conclusion needsto be conveyed to the remotely located patient. Notwithstanding, fasterbandwidth links are advantageous, and the use of the somewhat moremodern space-time coding technologies such as multiple-input andmultiple-output (MIMO) transmissions and its variants appropriate forthis invention will be apparent to those skilled in the art.

FIG. 7 summarizes the physical flow that creates the images finallyshown, without reference to the guidance system. An emitter 711 createsultrasound pulses which echo from structures 701 in the patient 700 andare detected by a transducer 712, converted to digital signals by ananalog-to-digital converter 720, compared by an echo identifier 730 withthe emitted pulses to detect echoes, and interpreted 740 as spatialreflectivity data by interpreting the echo delays as distances, andhence as 1-dimensional images along particular lines 530 from thesensor. In a particularly secure embodiment, these linear data arecombined in the cloud 750 (outside the local environment 766) withpositional information transferred 719 from the position sensor 717attached to the emitter 711 and transducer 712, to give a 3-dimensionalimage made available to the remote clinician by a display 760. It is notrequired that even 2-dimensional images exist within the localenvironment 766. (Indeed, the echo data could be transferred directlyfrom the echo identifier 730 to the cloud 750, but for bandwidth reasonswe prefer to perform a limited spatial interpretation 740 locally.)

More briefly: A method of acquiring medical images (understood asincluding printed, hard-copy, visually displayed images on CRT, LED,Liquid Crystal or plasma devices, or datasets for automated evaluationof fetal metrics, severity of kidney stone disorders, or other uses)arranges that the operator acquires clinically useful images byfollowing geometric criteria which can be verified by local computation,without reference to the images themselves for guidance. The computerdetermines these criteria with. reference to fiducials placed on thepatient, according to a standard protocol for a particular condition(such as pregnancy) and displays geometric feedback on the device'smotion sufficient for the operator to meet the criteria. The method hasan on-site user manually operate a 3-dimensionally tracked non-invasiveimage device to produce image data. A computer (preferably on site)determines whether the motion revealed by the tracking implies that theacquired data are sufficient for the reconstruction of a 3-dimensionalimage array appropriate for further visual or automatic analysis, suchreconstruction being in a preferred embodiment performed by a distalcomputer to which the image data are transmitted. Optionally, thecomputer reports to the on-site user an indication of image qualityperformance determined by the computer.

The apparatus and method may be used in training and in actualdiagnostic practice in the field. The training and the field operationwill be separately described. The description here uses obstetrics as anexemplary field of application: other fields of application are enabledin a manner that will be obvious to skilled personnel.

In contrast to the current training method described in the Backgroundand illustrated in FIGS. 1 and 3, in the system proposed by the presentinvention the training is performed according to the information flow inFIG. 4. It uses a geometric 3D display unit 405, rather than the 2Dacquired image display on the monitor 305, and an expert trainer is notrequired (though supervision and advice by a more experienced person maybe helpful, and is not excluded). The components in common with theconventional method are shown with similar numbers to FIG. 3, e.g., 413for the computer 313 and so on. The patient in the conventional schememay sometimes be substituted with a 3D anatomic model, includinginternal structures: the present invention requires only an externalresemblance, with the addition of fiducials and a tracking system, asfollows.

The scanning device 402 is equipped with a 6DOF tracker 450: a devicewhich reports the device's current location and orientation (roll, pitchand yaw, or equivalent information) in a ‘frame of reference’ or systemof coordinates (u,v,w). (Together, we call the location and orientationof the device its position.) Such trackers are well known to thoseskilled in the art, and may function in various ways (not shown): forexample, included accelerometers may allow computation of the currentposition relative to an initial state (as in a ‘Wii’ remote): thenecessary accelerometers, with software to derive positional data, areavailable separately from many sources, such as the ACC/GYRO, 3D, 16G,2000DPS, 28LGA 6DOF Analog Inertial Measurement Unit (3 AxisAccelerometer+3 Axis Gyro) manufactured by STMicroelectronics inBangalore. Alternatively, the tracker 450 may receive from a fixed basestation electromagnetic signals that allow computation of the currentposition relative to the base station: the necessary hardware andsoftware for integration into a system such as the present invention areavailable separately from many sources, such as the AuroraElectromagnetic Measurement System manufactured by Northern Digital Inc.in Waterloo, Ontario. Alternatively, one or more cameras may acquireimages of targets incorporated in the tracker 450 that allow computationof the current position relative to those cameras: the necessarytargets, cameras and software for integration into a system such as thepresent invention are available separately from many sources, such asthe family of products manufactured by Iotracker™ in Vienna. Any devicethat obtains such information may be used within the spirit of thepresent invention. The tracker may be integrated with the design of thescanner, or a scanner already on the market may be incorporated into acasing that also supports a tracker. Modern examples of such scannersfrom major manufacturers such as GE, Siemens, and others can easily beobtained.

The fiducials are small objects or tracker detectable inks or markingsfor which the tracking system may be able to detect the location, aswell as that of the moving scanning device 402, in the same (u,v,w)reference frame as is used in tracking the scanner: alternatively, theoperator may 408 touch each of them with the tracked scanning device402, in a prescribed order, indicating to the system by a click or othersignal that the current device position defines a fiducial position.Their orientation data is not required. At least three such staticfiducials are placed on the model patient, in anatomically standardizedpositions such as (by way of example) the navel, the anterior superioriliac spines, the pubic symphysis, prominent skeletal features, etc.;mostly skeletal landmarks that can be located by touch, with littletraining.

These positional data are reported 411 to a computer 413, perhaps fromintermediaries which compute them from raw data, such as a base station,a computer connected to cameras, a separate program running within thecomputer 413, etc. Any means that produces equivalent result positionalinformation may be used within the spirit of the present invention. Thecomputer 413 creates a 3-dimensional image which it transmits 410 to thedisplay unit 405 which is visible 415 to the trainee 403, who modifies423 the position of the tracker 450, attached to the scanner, which isagain reported 411 to the computer 413, and so on. It is desirable thatthe latency in this cycle be short, with a delay of the order of a tenthof a second or (preferably) much less.

The computer 413 could be a smartphone, tablet, or other computationaldevice already available to the operator, or it could be or couldinclude a special purpose chip which augments the said computationaldevices. The display unit 405 may be incorporated in the computer 413,as in a smartphone or tablet, or a separate unit, for increasedresolution and field of view. In our preferred embodiment theorientation of the displayed image corresponds to that of the dummymodel 412, and its location is near the same line of sight from thetrainee's perspective as in viewing the model 412.

It is desirable to the invention that the image displayed is3-dimensional, in the sense that it provides sufficient depth cues (suchas one or more of occlusion, directional lighting, perspective,parallax, etc.) to be perceived by the trainee in three dimensions,rather than as a flat image. In our preferred embodiment, the image isdisplayed in stereo, thus including the specific depth cue of presentinga distinct view to each eye, from which normal human vision cantriangulate distance. Such a presentation may use shutter glasses, whichalternately blank the view of each eye in synchrony with a high-speedpixel display which alternately show the views intended for the left andright eyes. These, together with the necessary synchronization hardwareand software, are available separately from many sources, such as the 3Dsystem for both cinemas and small screen use manufactured by XPAND inLimassol, Cyprus. Most shutter systems can use any pixel-orienteddisplay device, whether it is a panel of light-emitting diodes orphosphors or other digitally controllable light sources, a panel ofliquid crystals selectively filtering a backlight source, a projectorthat casts a digital image onto a passive screen by similar filtering,or other such devices standard in the display industry, provided onlythat they support a high refresh rate (120 images/sec is desirable) asdoes the graphics board, which must also support the synchronizationsystem. Such systems are widely available, with a preferred versionmanufactured by Nvidia in Santa Clara, Calif., integrated with itshighly parallel graphics processing units.

Alternatively, the invention may use passive glasses, with the left andright eyes seeing through complementary polarizing filters (e.g.,linearly polarizing in vertical versus horizontal directions, orcircularly clockwise versus anti-clockwise). The two views are arrangedto arrive with the corresponding polarizations (at a refresh rate on theorder of 50 or 60 images/sec) from the display. This arrangement is mosteasily achieved in a projection system, where the left and right eyeviews are projected from standard projectors simultaneously throughdifferent passive filters, or alternately through a synchronized filtersuch as in the projection technology made and sold by RealD Inc., inBeverly Hills, Calif., widely used in cinemas. It has more recently beenachieved at the display panel level by such methods as the systemsintroduced by manufacturers including TV makers LG and Vizio, withhorizontal polarizing stripes overlaying the screen, though this halvesthe resolution available to each separate eye. With prices geared to theconsumer market, this technology is incorporated in devices such as theMSI CX620 (MS-1688) 3D laptop. An embodiment of the present inventioncould use either such a laptop, or a separate screen obtained directlyfrom the manufacturers.

Alternatively, the invention may use an autostereoscopic display, usablewithout glasses. Each pixel location in such a system creates at leasttwo RGB color combinations, separately visible from different regions ofspace, so that if each eye is in a different region the eyes seedifferent images: this requires the eyes to be approximately inpre-planned positions, since with current technology the regions cannotbe changed on the fly. (More complex systems, with more than tworegions, allow more than one pre-planned ‘sweet spot’ for the user'shead.) Commonly, such displays use a system of small lenses (oftenvertical stripes) or a parallax barrier (with narrow vertical gapsthrough which the user sees different pixels form different directions)to confine the light from neighboring columns of pixels toward thetarget regions. A typical autostereoscopic system is the display of thelaptop Toshiba Qosmio™ F755-3D290. Again, the present invention couldincorporate either such a laptop or a separate display.

Alternatively, the invention may use a holographic display system,emitting a different view at every angle (not just at a small set ofangles, like a lens-based autostereoscopic system). Holography isclaimed to be used in the HoloVizio™ system manufactured by Holografikain Budapest, Hungary, perhaps in the manner disclosed in U.S. Pat. No.5,801,761, Method and apparatus for displaying three-dimensional images,to Tibor Balogh, 1998. A real-time system for holographic display to alimited number of tracked eyes has recently become available fromSeeReal Technologies S.A., of Luxembourg. We mention this approach herefor thoroughness, though it is unlikely that holography would becost-effective for a first embodiment of the present invention.

Alternatively, the invention may use the ‘anaglyph’ system with adifferent color filter for each eye (often red and blue) first describedin 1853 by Wilhelm Rollmann, “Zwei neue stereoskopische Methoden,”Annalen der Physik 166: 186-187. The anaglyph scheme is the most robustand least costly, requiring only a standard color display with paperglasses, and completely adequate for the present purpose (particularlyif the filters are chosen to work well with the common forms of colorblindness), and is thus our currently preferred embodiment.

All these methods of providing the stereographic depth cue can becombined with the parallax depth cue, where the visible image changeswith the motion of the head and thus of the eyes. True holographyalready contains this cue, by emitting a different view in everydirection, but the other displays can gain it (for a single viewer) byeye-tracking and computing the view shown using the current point(s) ofview of the user or of the user's separate eyes. This is described inthe paper C. Zhang, Z. Yin, D. Florencio, “Improving Depth Perceptionwith Motion Parallax and Its Application in Teleconferencing,” MMSP,2009, and could optionally be included in the present invention, thoughit requires additional hardware (one or more cameras observing the eyes,or a trackable head mounted device), substantial local processing power,and high speed in all elements of the system to avoid distractingdelays. This is an area of active research, but not yet robust enoughfor a preferred embodiment of the present invention.

Since these various depth cues are sparse or lacking in the figureformat used here, FIG. 5 presents the displayed view in a stylizedfashion.

The 3D display shows the fiducial positions as markers 501, using atransformation T that rigidly maps (u,v,w) locations around the modelpatient to positions in the display coordinates (x,y,z), where typicallyx varies horizontally across the monitor, y varies vertically across it,and z increases in the direction away from the viewer. The markers 501are shown optionally with (or as vertices of) a frame 503 (e.g.,graphical) displayed to serve as an additional depth cue for theirposition and for the additional display elements described below. Forclarity FIG. 5 shows four such fiducials in a square, and the frame 503as a cube, but other geometrical choices for their number and layoutwill be apparent to those skilled in the art. The line 530 illustrates atypical path along which an ultrasound pulse travels and returns, givingrise to echo data: a system using a single such path is an ‘A scan’,whereas the assembly 550 of such lines is a ‘B scan’.

Within this contextual display, and usually entirely within the frame503, the computer shows a ‘volume of interest’ (also referred to hereinas region of interest, often abbreviated VOI or ROI in literature) 505,corresponding via the transformation T to a volume V containing theregion in the model which holds anatomical structures of clinicalinterest (or would hold them, if the model were anatomically complete.Additional graphical elements such as lines 511 relating the volume ofinterest 505 to the frame 503 may be included as further depth cues tothe user. The computer determines the volume of interest 505 by codethat refers to the positions of the fiducials (assumed to be in theirstandard anatomical locations), and estimates the positions of thestructures of clinical interest. In a preferred embodiment the codemakes deductions from the fiducials regarding the posture of thepatient, the scale of the pregnancy (not identical to the developmentalstage, since the size and even the number of equally developed babiesmay vary), and similar variable elements, using these date to sharpenthese estimates and reduce the required size of V. In simple examples, Vcan be the convex hull of the fiducial points, or ‘all points within aradius R of the centroid of the fiducials’, or other such geometricspecifications adapted to the typical anatomy, but in a preferredembodiment, the system and software can acquire the needed completenessof scan from an ‘expert system’ (a term with a defined meaning in thefield of software and artificial intelligence). It can assess thenecessary sweep, to ensure that the volume of interest has been scanned,from a warping algorithm that maps the volume in a standardized, stageof pregnancy-adjusted anatomy to that of the individual mother-to-befrom obtaining the position of the fiducials placed on the externalabdomen.

In the same 3-dimensional image, but drawn in FIG. 5 separately forreasons of clarity in a limited format (and drawn for depth clarity froman angle which may differ from the viewpoint actually used in computingthe display), the computer displays a planar structure 550. Thiscorresponds by the same transformation T to the region in the patientmodel for which the scanner (if currently active), held in the currentposition, gathers material data by echoes. In training use it need notbe active, since the system is not training the operator to guide thescanner by display of the current scanned image: indeed, a ‘dummy’scanner with only the tracker functionality could be used for trainingsystems. Further embodiments may display a virtual or real anatomypreviously acquired, but no current acquisition is necessary.

In a preferred embodiment, the 3D frame 503 is so arranged relative tothe eyes of the trainee 403 that the transformation T involves verylittle rotation, so that the representation 550 of the acquisitionregion turns about approximately the same axis as the scanner unit inthe trainee's hand: a similar arrangement applies to the active imagecreation configuration described below. This harmony between therotations felt in the hand and the rotations seen in the display makescontrol substantially easier.

The training then includes learning to sweep the planar structure 550smoothly so that it passes through every point of the volume of interest505 with a speed and uniformity of scan consistent with acquiring imagesinterpretable by skilled personnel, and which yields the desired metricseither from automated machine processing of the image, or by interactionwith a human professional.

It is emphasized that no image needs to be acquired in training: thetraining module merely tests whether the volume of interest 505 issufficiently covered, and gives feedback to the trainee on this test.This is enabled by the said input reported (FIG. 4, element 411) fromthe 6DOF tracker (henceforth, tracker) 450 to the computational device413 in FIG. 4. Algorithms and software are used in computing theparticular scan area available from the location and orientationinformation as well as the accelerometer information on the smoothnessof the scan. In particular, there is no need to acquire any imageadapted to a particular anatomical orientation (which with a fetus,could be very variable), since the information for a 3D reconstructionis present in the invention. The approach currently used in the art doesrequire real time display of the images, plus the expertise to graspthem and to perceive in anatomical terms the current position of theacquisition shape S: gaining that expertise is a training bottleneck.The present invention makes the task, and training in the use of thesystem and methods, much easier.

To provide the information to the operator during training to ensurethat the image is properly acquired, smoothness of scan can be assessedby numerical tests of the succession of positions reported by thedevice, requiring changes in location and orientation to lie withinprescribed limits. The software does not perform the sweeping motion ofthe sensor, which is controlled by the hand of the field operator, butit can detect jerky or over-fast motion and call for a re-do. In anembodiment, this may be displayed as a clear jump in the scan region, byfor example showing the separated consecutive positions 550, togetherwith a suitable warning display.

In active practice, after training, the model is replaced by a patient,and data are actively acquired by the scanner. FIG. 6 shows theinformation flow of an embodiment where the elements in the logical box655 are the same as in FIG. 4. Thus all the numbers 402, 405, and so onare replaced by the corresponding numbers 602, 605 etc. which have thesame meaning as the corresponding numbers in FIG. 4 except that themodel 412 is replaced by a patient 612, and the trainee 603 is now anoperator. There need still be no means for the operator to see the imagedata collected, though the system may optionally be capable of this,either in a separate display or mapped 610 as a texture onto the planarstructure 550 within the 3D display 605, as seen 615 by the operator.(If this is done in the embodiment selected for practice, it can beanticipated in training by using a stored array of intensity levelsI(i,j,k), perhaps from a library of several such arrays, withoutrequiring the presence of real internal anatomy.) In a preferredembodiment these data are not displayed to the operator, but transmittedvia the uplink 670 to a separate site 690, optionally via temporary orpermanent storage in the cloud 680 and retransmission for delivery 685.At some point in this flow (the on-site computer 613, computing hardwarein the cloud 680, or the clinical display site 690), the echo data andposition data are reconstituted into a 3-dimensional array of intensityvalues, reflecting material properties at corresponding points in thevolume V, as performed for example by the Freehand™ 3D system describedin the Background above.

The construction of the 3D scan makes unnecessary the task of acquiringa 2D scan at a particular orientation, which is critical when using anultrasound scanner ‘live’, or when viewing only individual images. A setof 2D scans from an untracked scanner (as used in current practice) maybe useless in computing a fetal dimension metric like femur length, whenone end of the femur appears in one scan, the other end in another, andthe geometric relation between the scan positions is unknown. For suchpurposes, therefore, the user of an untracked scanner must selectindividually appropriate scans, whose quality can be judged only byinspecting the scans as they are made. The user of a tracked scanner, asdescribed in the Background, is in current systems guided by theimmediate display of the acquired planar image. The present practicethus absolutely requires anatomical awareness and judgment, and rendersinevitable that the operator must see what is acquired. If the criterionis only fullness of sweep, with many orientations equally acceptable,the task becomes simpler, and training not merely becomes faster, butcan be performed partially or completely with an automatic program, asin learning a video game without an expert tutor: this reduces the costof the system, and removes a bottleneck to rollout.

At the site 690 a clinical user (who may be remote, and work in acontrolled and licensed setting with institutional safeguards againstbreach of patient privacy, sex determination, or such other acts as maybe forbidden by local law) can see the reconstructed 3D data, usingmeans selected from the great variety of rendering and interface toolsdeveloped in recent decades. The anatomy can be rendered in 3D, makingit easy to identify (for instance) the ends of the femur for calculationof its length. The commands available in OpenGL and similar frameworksmake it simple to extract a planar slice in any position, and view it bya variety of user interfaces. Alternatively, or in combination with suchdisplay, smart software can extract such clinical parameters as femurlength. In a preferred embodiment, with or without input by a clinician,the system assembles the data into a report listing such parameters andother indicators of the state of fetal health, which is returned via thecloud to the operator. If there is cause for action in the report, thisfact is highlighted and the operator refers the patient to a localmedical practitioner, to whom the report is digitally or manually madeavailable. The local practitioner thus gets the benefit of an expert, orexpert system, analysis of the patient's and fetus's condition, while nolocal person needs to see the ultrasound images. If the social and legalenvironment makes it appropriate, the reporting format may specificallyexclude any way of specifying the sex of the fetus: moreover, the imagedata may be anonymized, so that a local person cannot successfullycontact a person at the clinic with a request to know the sex of aparticular fetus. The overt identity codes for patients, visible in thepatients' locality, and the IDs under which the image data are presentedand analyzed, exist in a hashed correspondence like that of usernamesand passwords. Many ways of structuring such security will be apparentto those skilled in the art.

Certain environments, as mentioned above in the Background, may mandatenot merely the local non-display of the images, but a guarantee thatthey remain inaccessible. The echoes received by the scanner do not inthemselves constitute images: they require substantial processing toproduce an Intensity (i,j) array of the kind that can be displayed aspixels on a display device. It is therefore useful to examine the flowof this processing (FIG. 7).

For clarity we show ultrasound emission 711 and the conversion by thetransducer 712 of received ultrasound to analog electrical signals asseparate functions, though typically they are handled by the samehardware. The emitter 711 creates emitted pulses 707 of ultrasound whichtravel to the region of the patient 700 to be scanned, along paths suchas the straight line 530. Reflecting from structures 701, the echoedsound 708 of these pulses travel to the transducer, from which analogsignals 715 travel to an analog-to-digital converter 720, and from thereas digital signal 725 is sent to the echo identifier 730 whichidentifies echoes and their time of flight. To do this the echoidentifier 730 absolutely requires transmission timing data 721, whichgive the times of emission, to travel from the emitter 711. The traveltimes cannot be estimated without this. For an intrusive device tocapture sufficient data in any stage up to the echo identifier 730,therefore, it must capture them at multiple points. A device which readsthe emissions and echoes, distinguishing them from each other andambient noise, then computes for itself the echo times, would requiremuch more engineering effort than making a stand-alone ultrasoundscanner (or buying an illegally trafficked one), with errors added bynoise and mis-estimation, and would legally constitute a scanner withthe present system as ultrasound source, so it is barred both by law andpracticality. It offers no advantages over an ordinary scanner, becauseit would show only degraded versions of the individual 2D scans: withoutthe 3D reconstruction function enabled by 6DOF tracking data, onlyanatomical expertise can offer the guidance necessary to acquire usefulscans.

Electronically capturing both the transmission timing data 721 and thetransmission of the captured echo analog signal 715 or as digital signal725 would require invasive electronic hacking at the hardware level,with no simple points of access if the functions involved arecommunicating within a single integrated chip.

As an additional security option, the digital communications 719, 721and 725 can be encrypted within the hand-held sensor unit itself, usingencryption hardware included within the device and a key specific toeach individual device, so that capturing the binary data alone has novalue. Many means of encryption will be evident to those skilled in theart: without limiting intent, we observe that these include theencryption of data from the emitter 711 relating times at which pulseswere emitted to the directions of these pulses (without which the line530 along which an echo returned cannot be known), making interceptionof the transmission timing data 721 valueless; the returned signals inthe form created by the analog-to-digital converter 720 may beencrypted, making interception of the digital signals 725 valueless (forthis method, as with the time/direction data from the emitter 711, theecho identifier 730 must be equipped for decryption); the data from theposition sensor 717 may be encrypted, preventing its use in constructinga 3-dimensional image from the 2-dimensional scans of successivepolygonal regions 555; the association between lines 530 and the echopatterns along them, analyzed by the echo identifier 730, may beencrypted, preventing their use in constructing even a 2-dimensionalimage; and so on. By moving any decryption step outside the process 766performed locally, an embodiment can prevent the existence of detectableimages within the local system, where human security (against bribery orother forms of persuasion) may be weak.

If the time-stamping of emissions from 711, the digitization by theanalog-to-digital converter 720 of returned signals, and the echoidentification by the echo identifier 730 are all performed within asingle chip, there is no vulnerability of the scans up to the point ofthe echo analysis by the echo identifier 730. The data flow 735 from theidentification of echo flight times and directions to the creation 740of an array of pixels need not pass along a distinct, hackablecommunication pathway between physical components, as it is mostefficient to use each evaluated echo within a single chip to modify oneor more pixel values in the image array, then discard it from memory,rather than store or communicate the whole set of such echoes alonglines 530. Access to data that a microchip does not send to itscommunication ports is practical only in a large and extremely costly‘clean room’ laboratory setting, with microscopic equipment. This doesnot guarantee that it will not occur, but it is evidently notcost-effective for the criminal. Moreover, if the legal clinical display760 is at a remote site, the unit is necessarily registered with alarger ecosystem that transmits 755 data and reports, manages andencrypts records, etc., and should include regular inspection of remoteequipment. It is straightforward to mandate, if legally required, thatsuch inspection does take place and includes alertness to evidence ofhacking at the hardware level.

Hacking at the software level, which need not leave physical evidence,can thus only occur once reconstruction processing has created a pixelarray, or equivalent data. It may be required that we guarantee theimpossibility of access to these data.

If reconstruction processing is deferred to the cloud 750, no imageactually exists within the local flow 766, so that access requires theaddition of local processing. Unless the software is locallyprogrammable, the hacker must add hardware, which will be easilydetected in the inspection suggested above. There is no need in a matureproduction device (as distinct from experimental prototypes) for localprogrammability, so the code can be unchangeably ‘burned in’ to theembedded chips. This thus represents a high level of security againstlocal access to images.

If the data size of unprocessed digital ‘time and echo’ data issignificantly larger than that of the reconstructing images, however,this deferral may not be optimal. It can coexist with relatively lowbandwidth, since the system does not require real time upload 745 to thecloud 680 or, as in this Figure, 750, (data can queue up to be sent overseconds or minutes, if there is local memory to wait in), but it impliesa significant cost both in total packets transmitted and in turn-aroundtime before a patient receives a report. In this case the algorithm usedto create the image array can be combined with a strong encryptionalgorithm, of a kind well known to those skilled in the art, using a keythat is specific to the particular unit and registered with the overallsystem in the cloud 750. In this case the image does exist locally, butnever as a whole in unencrypted form. Usable access to it would requirenot only physical access to the image at some point where it is storedor transmitted, but decryption of a level that uses special equipmentand expertise available only to large governmental and otherorganizations is easily available and known to those skilled in the art.Further, transmission via the web to the cloud can use a protocol suchas HTTPS (technically, the result of simply layering the HypertextTransfer Protocol (HTTP) on top of the SSL/TLS protocol, thus addingsecurity capabilities to standard HTTP communications) widely used forsecure commercial transactions. Similar encryption may be applied to thecommunication 722 of the tracking data for the locations and rotationsof the emitter 711 and transducer 712, which makes possible thereconstruction in the cloud 750 of a 3-dimensional data array to bevisualized in the interactive display 760, and is hence essential to theavoidance of a need to produce individually useful, anatomically guidedplanar images. Without these data, and reconstruction software that usesthem to turn a freehand series of slices into a volume data set that canbe sliced in new directions, the user of a hacked instance of thepresent invention requires the same level of anatomical expertise as theuser of an untracked scanner obtained on the black market.

By these or other arrangements that will be apparent to those skilled inthe art, it is clear that images can be made inaccessible to the localuser with a level of security adequate for presenting misuse.

In the context of legal prohibitions it will be necessary to considerthe working of the system in the light of the exact phrasing of the lawin each relevant country: the working may be slightly adjusted toharmonize with the law, or (in light of the evident social value ofmaking the medical benefits of ultrasound available without sexdetermination) the government may be persuaded to adjust the relevantActs or regulations.

To the best of our current knowledge, the use of 3D graphical display toguide the user's creation of the series through a geometricallydisplayed target volume is new, for either training or active use. Itsimportance here is to fully separate (as Freehand™ 3D does not) theselection of separate viewing-slice selection from the reconstructed 3Ddataset (as well as volume rendering, recognition of the whole femur bysmart software, etc.) from anatomical guidance of the original sliceacquisition, both making useful a series that was not guided byanatomical awareness to create individually significant slices, removingthe anatomy-based element of the required skill and training inacquisition, and creating the potential for remote medical viewing ofimages that need not locally be visible, or even (in the chosen sliceposition) exist. Although 3D reconstruction from hand-guided planarslices is not in itself a novelty, it is a further advantage of thepresent invention, that the reconstruction software need not evenrequire additional computing power locally (in the device or an attachedcomputer) since it can run on a server in the cloud, shared between manysuch devices. An embodiment may perform the computation either locallyor in the cloud, depending on the trade-off between the cost of localcomputing power (which for decades has dropped according to Moore's Law)and the cost of communication bandwidth. In either case, theavailability of 3D images for algorithmic quantification of diagnosticmeasures permits improved diagnostic value without the requirement formore costly 3D-specific hardware such as a parallel beam device.

Recapitulating, an aspect of method technology in the practice of thepresent technology includes a method of acquiring medical images, inwhich: a) an on-site operator manually operates a non-invasive sensingdevice to acquire echo data from points at computable distances anddirections from the device, interior to a chosen region of a subject(e.g., at internal body locations where organs or tissue growth occurs,including a fetus or tumor); b) identifying fiducials placed atanatomically defined exterior points of the subject; c) determining thelocations of the fiducials in the reference frame of aposition-determining subsystem; d) establishing a correspondence betweenspace physically occupied by the fiducials and numerical space in whichthe chosen region is specified; e) monitoring the position andorientation of the sensing device at the time of each acquisition of aset of echo data; f) optionally preventing local access to such echo inany form which would allow local reconstruction of an image; g)displaying by computer on a visual display device a combinedrepresentation of the positions of the fiducials or of a referenceanatomy model located in reference to them, of the chosen region, ofeach successive current location concerning which the sensor can obtainecho data (in a preferred embodiment, without the echo data themselves),with the sensor in a current position reported by the position andorientation monitoring sub-system; h) the computer determining whichsub-regions of the chosen region have not yet been sensed in levels ofdetail and of quality that satisfy pre-determined quantitative imagecriteria, in respect of the acquired data; i) the computer displayingthe said sub-regions to the operator, showing the sub-regions inrelation to the displayed representation of the fiducials; j) assemblyin a computer of the acquired data into a three-dimensional image dataarray; and k) using the three-dimensional image data array to display animage to a user who need not be at or near the location where thepatient is scanned.

A further aspect of the present technology is the system enabled andconstructed to perform the methods, such as a system for acquiring andtransmitting medical images, having:

a) a manually operable, non-invasive sensing device capable of acquiringecho data from points in an interior region of an animal subject atcomputable distances and directions from the non-invasive sensingdevice;b) fiducials (markings that are at least one or both of visuallyobservable or mechanically observable) at anatomically defined exteriorpoints of the animal subject;c) a computer configured to determine locations of the fiducials in areference frame of a position-determining subsystem;d) the computer, in communication with the non-invasive sensing device,being configured to establish a correspondence between space physicallyoccupied by the fiducials and numerical space in which the interiorregion exists;e) the computer configured to determine and to monitor from receivedsignals containing echo data, position and orientation of the sensingdevice at a time of acquisition of each data point within a set of pointdata;f) a visual display device in data communication with the computer, thevisual display device configured to provide an image of a combinedrepresentation of positions within the region of the fiducials or of areference anatomy model located in reference to the fiducials, of eachsuccessive current location concerning which the sensor can obtain echodata, with the sensor in a current position according to the positionsensing sub-system;g) the computer configured to execute code to determine whichsub-regions of the interior region have not yet been sensed in levels ofdetail sufficient to meet predetermined standards of image qualitywithin the acquired data;h) the computer configured to prevent local access to echo or imagedata, optionally including hardware encryption;i) the computer configured to transmit image data from the acquired datato display the sub-regions to the operator on the visual display device,displayed images showing the sub-regions in relation to representationof the fiducials;j) a computer configured to assemble the acquired data into a3-dimensional image data array; andk) connection to a system for display of such 3-dimensional image data.

It is an aspect of the technology of the invention that the displaypresents the operator with a three-dimensional view. For purposes ofclarity, ‘three-dimensional’ is a term with broader meaning than acommon interpretation as ‘stereographic’ which refers to presenting theviewer's two eyes with two distinct views, from which triangulation inthe viewer's visual cortex can deduce distance. (There are many ways toachieve such a presentation, including but not limited to the wearing ofglasses with different color or polarization filters ordisplay-synchronized shutters for the two eyes, the use of prisms togive two displays the same apparent position, the placing of differentviews directly in front of the eyes with lenses to adjust focus,holography, Pepper's ghost displays and other technologies familiar tothose skilled in the art. All may be used within the spirit of thepresent invention.) Popular usage of ‘3D’ tends to refer narrowly tostereography, but for present purpose ‘3D’ or ‘three-dimensional’display refers to any display in which there are depth cues that can bevisually or automatically interpreted as having width, height and depth.Using FIG. 5 as itself an example of depth cues, the overall frame 503which is recognizable to a viewer as a rectangular box, having widthacross the figure, length vertically in the figure, and depth ‘into’ thefigure, despite being in itself a two-dimensional pattern of dark pointsagainst white ones. This is achieved in this instance by arepresentation of occlusion by the breaking of lines that are to beperceived as passing behind other lines, as in the breaking of all linesthat pass behind the edge 504 of the frame 503. (With wider objects indiffering colors, this graphical device of a visible break in theoccluded element is not needed.) Such cues may include but are notlimited to occlusion (illustrated by the way parts of the frame 503 passin front of the polygon 555), perspective, and parallax: the way that aneye's view changes with the eye's motion. This cue may be included inthe present system either by an eye-tracking camera and software(increasingly available as a commodity item) or by other means (e.g.,automatic tracking through software or manual response to softwareindicators) of tracking head position. Since the proposed system alreadyincludes tracking hardware able to continuously report the location ofthe scanning device 202, and such hardware can often follow more thanone object (for example, the FASTRAK® hardware and embedded softwaresystem sold by Polhemus tracks up to four sensors), inclusion of headtracking for parallax is a natural though not essential variant of theproposed system. Other cues to precise 3D location include the additionof graphical elements such as the lines 511 (without which a single-eyeview like that in FIG. 1 does not show directly how close the region ofinterest box 505 is to the near side and top of the frame 503), such asrendering with shadows cast by one structure on another, and such asothers well known to those skilled in the art. Any one or more of thesemay be used in the spirit of the present invention to qualify thedisplay as ‘three-dimensional’.

This invention discloses various additions to the display and datareconstruction previously disclosed. We refer below to the region 112acquired, by the same identifier, even when the sensor is used as partof a system arranged differently to FIG. 1. Recall that the system, bycontaining anatomical information, can predict with reasonable accuracythe location of internal organs such as the uterus, heart or gallbladder. These points are referred to as ‘fiducial points’, or morebriefly ‘fiducials.’ Their positions may be identified automatically bythe system itself, from a visual inspection comparable to the operator'sview of the subject, but in our preferred implementation the operatorplaces small physical objects stably on the subject's skin, at thefiducial points. (By a common elision, these liducial objects' are alsoreferred to as ‘fiducials.’) The positions of these objects, in thecoordinate frame of the position-reporting device, may be established bythe tracking hardware and software system acting without human input,for example by inclusion in the objects of position sensors such as themultiple sensors supported by the FASTRAK® system, or of visual featuresmaking them easily located by a camera-based system, or of other suchfactors appropriate to the tracking system. However, in one preferredimplementation the fiducial objects are simple passive items, which theoperator may touch in turn using the tracked sensor. This is discussedin more detail below, with reference to FIG. 11.

Data most directly obtained by an ultrasound device are notpoint-by-point images (such as the collection of numerical pixel valuesobtained and stored by a digital camera, or the brightness at differentpoints in an X-ray film, resulting from chemical changes triggered byarriving photons), but rather are echo records. An emitted ultrasoundpulse in a particular direction (comparable to an optical ray line,arriving at one pixel and modifying that pixel's brightness and color)is echoed by sound-reflecting matter at different distances, with theechoes received at correspondingly different times. The use of a singledirection, with a plot of echo intensity against time, is known as‘A-scan’ ultrasound. It is useful (for example) in measuring thedistance between the retina and lens of the eye, but the plot is not animage of the eye. An image of the eye along that line would consist of aline with varying brightness B, proportional to echo strength. For asingle line the plot view is clearer, but for a fan distribution oflines in varying directions, this ‘B-scan’ mode gives a useful display,combining to give a planar view of a slice through the eye or othertissue. This is the most common sonogram display in current use. For thepurposes of the present invention it is significant that the echo dataonly become pictorial by means of a non-trivial computation, essentially‘the echo along line L (such as the line 530 in FIG. 5: the polygon 555is a fan-shaped collection of such lines) has intensity I at time T,which corresponds by the speed of sound c to reflection at a point T/2calong L, passing near the space point we represent by pixel P, whosebrightness we increase by I, more strongly if the passage is nearer’. Incurrent art this computation is performed in real time and the resultingset of pixel brightnesses shown on a 2D display, but these values arenot the ultrasound data themselves. Moreover, if the direction of theline L is unknown, image creation is impossible. In particular, if thedirection is unknown to a person or system with access to the data, dueto encryption of data specifying it, that person or system cannotconstruct the image. In the present invention, it is not necessary toprovide the operator, or any local person or system, with thecoordinated system of directions, times and echo strengths that makeimage reconstruction possible. This permits an important enhancement ofsecurity in embodiments of the invention where image access is denied tothe operator, since locally no image even exists for unauthorizedaccess: the image-creation computation can be performed elsewhere, andthe created image data made available only to distant authorizedclinicians, who examine them in 3D or by slices chosen independently ofthe original positions of the hand-held sensor. A report of predefinedinformational content is returned to the patient site, optionallyincluding images selected by the clinician, which can exclude proscribedinformation such as sexual anatomy.

In a first method of additional guidance, the internal structure of thepatient (represented by a line hatching pattern in the subject 801) isscanned 802 by the operator 803 in the region 812 and the resultingimage is shown not as a polygon 810 in a fixed display 805, as incurrent practice, but as a texture polygon 555 mapped to the polygon 550which moves within the frame 503. This variant does not concealanatomical detail from the operator, and would not therefore beappropriate in markets where sex-selective feticide is a problem, but itdoes assist the operator in relating hand position to anatomy, and cantherefore in other markets offer better cognitive ergonomics. In somescanner positions the angle of view of the polygon 550 may be toooblique for clarity, but there is no bar in this variant to a separatedisplay polygon 810 in a fixed display box 805, 806 of the usual kind,in a separate display device 813 to which the probe is connected via alink 811 which may be wireless or in a shared window.

In a second method of additional guidance, the currently acquired imageis displayed in the same real-time way as a texture on the polygon 550,in the same apparent location, but it is blurred throughout. Forillustration within the conventions of patent Figures, we represent theimage data as a pattern of dots such as 901 or 902 in FIG. 9. (Currenttechnology uses shading, in gray levels and sometimes, to representadditional data such as Doppler values describing motion, in color.) Ifthe scanning device and processing is sufficiently precise, considerableanatomical detail can be seen, represented here by the contrast betweenthe patterns 901 and 902. Since this detail can include the sex of afetus, we may wish to suppress it from local displays. Without thecomputational difficulty of identifying anatomy, we may simply modifythe local display by for example convolving the image with a Gaussian:this is represented in FIG. 9 by randomly moving the individual dots inthe patterns 901 and 902 to create the patterns 905 and 906respectively. (Blurring may be done by such displacement of smallelements, or by pyramidal blurring, or by convolution with a Gaussiankernel, or by the fast method discussed in Quasi-Convolution PyramidalBlurring by M Kraus in the Journal of Virtual Reality and Broadcasting:6(2009), or by many other means well known to those skilled in the art.)The result is that large features such as the dark and light areas inthe patterns 901 and 902 remain visible, but the fine differences are nolonger apparent. In a fetal scan the wall of the uterus could remainblurrily visible, as would the location of the fetus, but fetal detailssuch as the genitalia would not. Blur would diminish the diagnosticvalue of the image, but the operator is not responsible for diagnosis.It applies only to the local display, while image data with allavailable clarity are transmitted to the remote clinical center.

We expect the visibility of large scale features even in a blurred viewto enhance the operator's skill and confidence in moving the scannerappropriately, particularly when they repeatably appear in a particularthree-dimensional location in the frame 503, assisting the operator increating a three-dimensional mental map of the subject's anatomy.

Such blurring may also be used with a traditional ultrasound display(FIG. 1), showing the blurred image in a planar image 106, incombination with transmission of the un-blurred data to a remotelocation, as disclosed in our previous application. However, ourpreferred implementation remains a three-dimensional geometric display,where hand-eye coordination makes it easier to judge the spatiallocation of the large-scale anatomical features which remain apparentthrough the blurring.

In a third method of additional guidance, the system finds edges in theimage data, by means familiar to those skilled in the art. (See, forexample, A Survey on Edge Detection Methods, Technical Report: School ofComp. Sci. & Elec. Eng., University of Essex: CES-506, M A Oskoei & HHu, 2010, or A Survey on Edge Detection Using Different Techniques, KKaur & S Malhotra, Int. J. of Application or Innovation in Eng. &Management 2:4, April 2013).

Commonly such means involve a first stage of detecting ‘edge-like’points by some measure of local contrast, including but not limited tothe use of operators such as a Sobel, Prewitt, Roberts, or Laplacian.Often many non-edge points appear edge-like due to noise, or edge-pointsfail detection due to local blur, so a second stage uses the detectiondata to construct substantial edge curves. This may be done by findingchains of edge points and discarding small fragments, but it is oftenmore effective to use active contours (pioneered in Snakes: Activecontour models, M Kass, A Witkin, D Terzopoulos, Int. J. Comp. Vision1:4), 321-331, 1988), which numerically move around and fit themselvesto as high a total of edge intensity as possible, while resisting sharpbends. This resistance may conceal detail, which for some applicationsis undesirable, but for the purposes of the present invention is abenefit, as is the reconstruction of a smooth edge from noisy imagedata.

By the above means, or by other edge-finding means familiar to thoseskilled in the art, we fit edge curves to the image acquired by thescanner. This is illustrated in FIG. 10 by curves 1001 and 1002 fittedto the patterns 901 and 902 respectively. In the present invention welimit display of these curves to long, low-curvature segments of them,as in the curves 1011 and 1012, which we show on the moving polygon 555or the fixed display 805, or both. For a segment to be displayed, itmust pass both a ‘length test’, such as restricting display of imagecontent having a length only in excess of 4 cm (a high upper bound forthe penis length of a full term fetus), and a ‘curvature test’ that theradius of curvature must nowhere be less than a predetermined value suchas 1 cm to allow display of the image content. These limitations rendernearly impossible the revelation of fine detail such as genitalia, whileproviding strong cues as to anatomical location of the region 112 thatis scanned. As with blurring, these partial edges may also be shown in atraditional ultrasound display (FIG. 8), showing them in a staticpolygon 810, in combination with transmission of the image data to aremote location. However, our preferred implementation remains athree-dimensional geometric display, where hand-eye coordination makesit easier to judge the spatial location of large-scale anatomicalfeatures.

Beside echo density, many other features may be extracted and displayedfor guidance, such as a sinus or similar cavity, within the spirit ofthe present invention.

In a fourth method of additional guidance, a preliminary analysis of theecho data locates points whose sound reflections are sufficientlystrong, relative to the other echoes currently received, to imply thatthese points are occupied by bone or a calculus such as a kidney stoneor gallstone. These points are added to the display, either within thecurrent polygon 550 or 810, or cumulatively within the three-dimensionalview, without revealing fetal sex but assisting the operator inanatomical judgment of the scanning location (whether fetal orotherwise). If the scan has a particular anatomical target within afetus or targeted organ, and the operator has enough anatomicalexpertise to recognize from the skeletal cue that this target hasalready been imaged, the operator may terminate the scan before thesystem reports that the target box is completely filled. Conversely, ifthe quality of the skeletal view appears poor in a particularsub-region, the operator may rescan that sub-region more carefully,indicating to the system that the new data are to be emphasized overthose acquired earlier.

In a fifth method of additional guidance, features such as those justdescribed are displayed 1021 or 1022 in superposition on the blurredimage 1025 or 1026 created as described in the second method above.

In a sixth method of additional guidance, numerical measures of clarityare applied to the image data. These may include, but are not limitedto, the localization of high values of edge-likeness measures, asdiscussed above; the value of entropy computed for the values assignedin small connected regions; the degree of correlation between valuesnearby in the image, as against values more separated; and other suchmeasures as will be evident to persons skilled in the art, within thespirit of the present invention. If high edge-likeness values occur forspatially large regions, if entropy values are above an appropriatelychosen threshold, if correlation values are below a threshold, etc., itis clear without human visual examination that the image isinsufficiently clear, and the image fails a ‘local clarity test’. Wherean image passes these tests, we may apply such ‘local resolution tests’as subsampling, supersampling the result, and assessing the amount ofchange. If this change is small, the local resolution of the image iseffectively that of the sub sampled (coarser version), which in thepresence of values differing by more than a threshold amount indicates alevel of resolution inadequate for the discovery of clinically importantdetail.

Another measure of quality relates to the fact that the acquisitionregion may be moving 550 may be moving slowly in the direction normal toitself, producing closely spaced planes of acquired data, but movingfast in a lateral direction, with resulting motion blur. This may bedetected by computing anisotropy measures of the correlation betweenneighboring data slices, with levels above a threshold failing thecorresponding ‘transversality test.’

We refer to tests of clarity, of resolution and of transversalitycollectively as ‘local quality tests’. Also included in this term is astraightforward test for gaps: if for a particular point in the targetregion, no image data have been registered for any location within apredetermined radius of that point, the point fails a ‘gap test’. Othersuch tests may be added, within the spirit of the present invention. Inparticular, the construction of such tests for Doppler data (reporting acomponent of velocity radial to the current position of the sensor) willbe evident to those skilled in the art.

We display in the frame 503 the region of such unsatisfactory pointswithin the polyhedral volume of interest 505 as a translucent blob, as acloud of opaque points, or as such other visible indication within thecapability of the available graphics processor as most clearly gives theoperator a sense of the spatial location of this region, and signal therequirement for repeat scanning of this region. Each type of failure isassociated with particular causes, such as: a low value of resolutionindicates that the user is moving the sensor too fast; a too-wideseparation of acquired data planes indicates a motion too fast of theacquisition region 550 in the direction normal to itself; failure of atransversality test indicates a motion too fast of the acquisitionregion 550 in the direction tangent to itself; a high level of entropyindicates jitter in the sensor location and/or orientation; and soforth. These may be confirmed by analysis of the series of recordedlocations and orientations of the sensor. Corresponding to each of theseproblem modes is an improvement that the user should make, whenre-acquiring data for the problem region: respectively, these may besummarized as the suggestions “move more slowly”, “turn the acquisitionplane more slowly”, “turn the acquisition plane more normally”, and“move more smoothly”. (Since recorded motion can be analyzed withoutreference to the acquired image data, similar messages can be includedin a training schema for the system, without any images being acquiredor processed. Users can thus become well practiced in the required styleof movement in an inexpensive imaging-free version of the system,omitting the image sensor and processing equipment, and sensing onlypositions and locations.) We refer to such messages to the user as‘feedback hints’.

When the operator has completed this task, and achieved adequate qualityin the region of previously unsatisfactory points, the resulting dataare three-dimensionally integrated with the others acquired in thissession, in creating a diagnostic data set for the remote clinician.

In a seventh method of additional guidance, numerical measures areapplied to the acquired image data, revealing whether their planes aresufficiently closely spaced for the reconstruction of a volume data set.If they are not, the region within the volume of interest 505 wherebetter data are needed can be three-dimensionally marked (by a smallerbox, or by other graphical means familiar to those skilled in the art)as requiring a repeat visit to the region 112 by the scanner.

In an eighth method of additional guidance, the ultrasound system iscapable of Doppler analysis, for example giving indications of maternaland fetal blood flow by frequency changes in the echo. Such analysis issubject to problems such as aliasing, where the low pulse repetitionfrequencies more sensitive to low flows/velocities result ininterference effects with periodic motion in the scanned tissue.Moreover, it is of the nature of Doppler analysis that it reveals onlythe component of target velocity toward or away from the emitter-sensorunit, not across that direction. To assess blood flow along an artery,using a single scan view, the sensor must be ‘looking along’ the artery,rather than seeing the artery from a sideways direction. (This languageis appropriately vague, since the sensor does not need to be preciselyin line with the artery. That case gives the strongest signal, but anangle of 30° between the viewing direction and the artery directionreduces the reported component by less than one seventh. Even an angleof 60° reduces it only by half) In a use like obstetric examination,where the fetus may rotate unpredictably, the best direction cannot beknown in advance.

In current practice, an experienced operator alters the scanningapproach to obtain good insonation angles so as to achieve unambiguousflow images, as judged by immediate viewing. In the system heredisclosed, the operator does not have a real-time view of the flowimage, by which to adjust for good quality. Consequently, just as forstatic anatomical use we replace the acquisition of individuallyrevealing 2D images by the acquisition of intensity data for a 3D region(later resampled in arbitrary planes, or volume rendered, or otherwisedisplayed independently of the original sensor positions), we acquire3-component vector data for a 3D region. One 3D scan of the kindpreviously described gives one flow-velocity component at each point,the component in the direction from the sensor. We thus require at leastthree such scans, so arranged that the three ultrasonic ‘rays’ passingthrough each point do so in mutually transverse directions, rather thancoplanar or collinear (or within a limit of, for example, 45° from thesedegenerate cases). This may be achieved by the operator moving thesensor around three non-overlapping zones of the body surface, whileaiming it at the same internal region of interest, or by other motionprotocols that will be evident to those skilled in the art. As analternative, sensors may be mounted on a separating frame and directedtoward a common three-dimensional region. This involves less operatortime (and a lower probability of target movement complicating theprocess) but more hardware and more operator skill, and is not ourpreferred first embodiment.

For each point p, the system computes a best-fit single vector for the3-dimensional flow vector v at that point, from the three or moreseparate echoes recorded from that point. (Note that the field ofvectors v is in our preferred first implementation not displayed to theoperator, as this would also reveal anatomy, but only to a remoteclinician. However, it may be displayed to the operator in animplementation for strictly local use, and its 3-dimensional characterhas advantages over the current art, for such use.) To a considerableextent the use of echoes from multiple directions can reduce the impactof interference effects and other artifacts common in a single-directionscan, but where the best fit v fails to come within an appropriatecriterion of matching the echoes actually observed, we mark p as a ‘badpoint’.

In a preferred implementation of the above eighth method, the systemdoes not merely detect that the Doppler data are inadequate, requestinga repeat, but derives a probable angle of attack for the scanner atwhich better results could be achieved, and gives an indication of thissuch as a visible three-dimensional arrow (or analogous indicator ofposition and direction, as will be evident to one skilled in the art),within the frame 503, to assist in achieving a better result.

A ninth method of improved acquisition of an integrated 3D scan allowsfor movement of the target during the scan, such as that of a fetus. Abest fit description of the motion during the scan of identifiably rigidelements such as fetal bones allows the creation of an overall4-dimensional velocity field, with a motion vector at points (x,y,z, t)representing position (x,y,z) and time t within the scan. By integratingthis to give motion curves, echoes from a point (x,y,z,t) can bereferred back to the spatial location of the same anatomical point at afixed reference time t₀. This computation is performed before presentingthe final 3-dimensional scan to the clinician, with removed or reducedartifacts of the motion.

A tenth method of improved acquisition of an integrated 3D scan allowsfor regular, repeated movement during the scan of the target, such asthe beating of the heat. This motion is unavoidable, but periodic ratherthan unpredictable. It repeats fairly accurately (in calm conditions) atintervals of T=(one minute)/(current heart rate), typically less than asecond. Collecting the scan data together with time stamps and theoutput of at least one electrocardiogram or other pulse-related sensor,and assuming regularity of the motion, we collect echo data for asufficient density of points (x,y,z,τ), where (x,y,z) represents spatialposition and τ is cardiac phase: the fraction of T elapsed since (say)the closure of the atrioventricular valve. The acquired image is thus4-dimensional (giving either scalar echo strengths or vector flow data),and the repetitive nature of heart motion means that gaps can berevisited. In an exemplary illustration of the guidance provided in thiscase, the display may show a square descending at a convenient speedthrough the region in which the heart is located, leaving the top of theregion at a time when τ=0. The operator is required to move the sensorso that the geometric display of the location of the data acquisitionplane moves with this square: it is not necessary to reach the bottom ofthe region at the same phase as it left the top. When this motion hasbeen achieved with sufficient accuracy, a square moves down with thesame speed but beginning at a time (for example) one-tenth of a periodlater, with τ=T/10. This is followed by a motion beginning withτ=2T/10=T/5, and so on for ten motions altogether. By this means,combining the capabilities of a human operator and consumer-levelelectronics, a basic planar sensor becomes a scanner which can supportquantitative estimates of such diagnostically valuable indicators asejection fraction.

Once the three dimensional image data set has been constructed from thecollection of multiple echo data points, the final set of threedimensional image data can be used in a number of different fields oftechnology. The image data may be used to indicate a need for surgicalor medication treatment. They may be used to numerically determine theshapes of scanned structures, such as a femur or heart, and derivetherefrom measures of clinical importance (see for example the paper“Segmentation of Speckle-Reduced 3D Medical Ultrasound Images” by P CPedersen, J D Quartararo and T L Szabo, 2008 IEEE InternationalUltrasonics Symposium Proceedings, pp 361-366, and references therein,for relevant art). The 3D image data may be used to determine thequantity of medication to be delivered, and be used to determine thepoint of delivery (e.g., by catheter or by hypodermic needle). The 3Dimage data may be used to define regions for specific surgical plans(including direction of access to the subject of the image data, e.g., afetus), enable the data to be exported to a three dimensional printingdevice to create models of the subject of the image (e.g., model of thefetus or the baby, physical models of the bone structure of conjoinedtwins, model of the organ structure within the patient), replication oforgan images for evaluation by surgeons, use of the organ structuremodel for training of surgeons or practice of procedures, production ofa holographic image from the three dimensional image, and other materialmanipulations using the three dimensional image data. These extendedprocesses are known in the art and the apparatus used to convert threedimensional data into a substantive product such as an image, plan orphysical structure is commercially available. These known technologiesare intended to be used with the commercially available apparatus andmethods, in the framework of the present invention.

In an eleventh extension of the earlier disclosure, thethree-dimensional image reconstructed is used for the automatic analysisof anatomical structures such as a growing femur, including quantitativecomparison with growth norms, objective sex identification to benon-erasably stored (permitting correlation of a clinical institution'srecords with later abortions, to raise statistical warning of securityleaks), identification of structurally important features, and forcertain conditions (such as identification of conjoined twins,mis-positioning of a fetus within the womb, or inappropriate arterialconnections in an adult or immature body) identifying plans oftreatment.

We now describe in a solely and non-limiting exemplary manner aprocedural flow of the use of this invention, with the understandingthat variants will be evident to those skilled in the art.

In a representative embodiment, the operator of the sensor is separatedby distance (hundreds of miles, or a few miles of city traffic) orinstitutionally (separate parts of a hospital building, with ‘Chinesewall’ arrangements in place). The operator need not have knowledge ofanatomy, as current sonographers do, with a required training period ofseveral years. We describe first an exemplary flow of events at the siteof the patient, the operator and the scan, in a case where the tissue inthe target box is static. The main non-static cases are the heart (whichbeats constantly, but regularly) and the fetus (which moves lesspredictably), as discussed below. Lung movement, when regular, may betreated similarly to the heartbeat, but we prefer to make the scan fastenough to be accomplished within a breath hold. Within the spirit of theinvention, the sequence may vary in minor ways evident to one skilled inthe art.

In FIG. 11, the local system 1100 first loads the identifiers of thepatient (name, patient ID, etc.), as entered into by an ElectronicMedical Records system (EMR), following principles well known to thoseskilled in the art, with a specification of the organ targeted for scan,and in some embodiments the scan type. (As described above, a Dopplerscan requires data collected from widely separated points, andechocardiography with this invention requires coupling the data analysisto the phase of the patient's heartbeat. For clarity here, FIG. 11 takesthe case of imaging static tissue.) The local system loads 1102 fromlong-term memory (local, or otherwise accessible) a simplified referencemodel A of human anatomy with the patient's gender and optionally suchcharacteristics as age and body mass index (BMI). Associated with eachoption for a target organ is a list of a plurality of at least M=3 offiducial points, which is loaded 1103 after the model A. In a displayallowing a sufficient set of distance cues to allow human depthperception to operate, the local system displays 1104 a simple view suchas a ‘wire frame’ surface of the anatomy model A, with the M fiducialpoints visually indicated. In a preferred embodiment, the first suchpoint to be physically identified on the patient is highlighted by abrighter color, flashing, or the like.

Optionally, a view of the target organ (within the reference anatomy) isincluded. The next step 1105 is performed by the operator, whoidentifies on the physical patient the points corresponding to thedisplayed fiducials. This may be done mentally, by visual inspection andfixing them in the mind, but in our preferred embodiment the operatortemporarily attaches physical fiducials at these points, since accuracyis more easily trained and achieved. If the position sensing function ofthe sensor has not been activated, this function is 1110 made active atthis stage, and the number N of entered fiducials set to 0.

There follows a loop, repeatedly testing 1120 whether all fiducials havebeen entered. If 1120 the number N is still less than M, the operatortouches 1122 the (N+1)^(th) fiducial point with a standard point on thehand-held sensor, and clicks 1123 a button on the sensor or a keyboardkey or otherwise signals to the system that the sensor is in positionfor that point. The system then 1124 records the location (by referenceto the coordinate system in which the position sensor reports) as thatof the (N+1)^(th) fiducial, and increases N by one. (Optionally, a stepnot shown may test for whether the operator has become conscious of anerror, and ‘recall that move’.) When N has increased to M, the loopconcludes. The local system then computes 1130 a best-fit transformationT between the coordinate system of the sensor's location-reportingsystem and the coordinates used in the reference anatomy model A. (IfM=3 exactly, the best fit is an ‘affine’ transformation, carryingstraight lines to straight lines. If M>3, the best fit may becurvilinear.)

Using the transformation T, the local system defines a polyhedral targetbox B in position-sensor coordinates, that can be expected to containthe target organ O, and 1131 initializes the box B as empty ofecho-derived data. The local system then 1140 displays athree-dimensional view that includes the wire-frame anatomy A(transformed by . into position-sensor coordinates), optionally othervisual depth cues such as a surrounding rectangular frame and a wireframe view of the expected position of the target organ O, derived fromthe reference model, and essentially the target box B. The local systemalso shows in the same three-dimensional view a polygon 550 representingthe zone Z from which the sensor is currently collecting echo data,deriving the position of that zone from the sensed position of thesensor. In our preferred first embodiment the zone Z is a fan-shapedplanar region, since the already-existing technology of collecting suchdata has been refined for this case, in order to show the derived imageon a planar display. However, a curved zone Z, or a zone Z filling athree-dimensional region of space as a torch beam does, falls within thespirit of the present invention.

A loop now begins as the system acquires 1141 acquires the ultrasoundecho data: the hand-held sensor emits pulses, detects the echoes as afunction of time. (At this point either the echo data or the positiondata or both may be immediately encrypted, but in the embodimentillustrated here we show encryption as a slightly later step 1165applied to the full scan's data.) The system's computer 1142 tests thedata quality, for example by evaluating the entropy of intervals withinthe data, relative to the overall variation in data values. High entropyvalues occur where the data vary in an effectively random fashion,independent of what is echoing them: a steady level of echo from auniform tissue, a jump between different values at a tissue boundary, oran isolated peak produced by an echo from a reflective membrane, producelower values. A similar measure is the degree of correlation of valuesfor particular times with the values for nearby times, as distinct fromthe values for times more distant. These are examples of what we shallrefer to as a ‘data quality measure’, with a corresponding ‘data qualitytest’ of the entropy value being below a pragmatically establishedthreshold, or the correlation being above a threshold: other suchmeasures and tests will be evident to persons skilled in the art, withinthe spirit of the invention.

If the data quality test is passed, the system stores the echo 1151 andsensor 1152 position data in conjunction with the sensor's location datafor that point in time. The system then 1153 uses the positional datafor the acquisition zone Z to label points in the box B that are on orclose to Z to as ‘filled’, without (in our preferred embodiment for theIndian market) completing the action of filling them with echobrightness values. In an exemplary manner of achieving this, the targetbox is shown as containing an open three-dimensional mesh, where nodesin ‘filled’ points are colored green (together with segments joining‘filled’ nodes), while the others are red. Many other ways ofrepresenting this information will be evident to persons skilled in theart, within the spirit of the present invention. The system then checks1160 whether any points in the box B remain unfilled. If some do, thesystem permits a small amount of time to pass, during which typicallythe sensor is moved 1162 by the operator (as part of a continuing smoothmovement, not a shift to a new static position), and 1140 updates thedisplay. If the box B is filled, the local system encrypts the data(unless this was already done, perhaps in parallel with the otherprocesses in the main loop 1140-1141-1150-1160-1162-640) and exports thedata set of echoes and corresponding sensor positions to the distributedsystem from which the local system 1100 loaded the case data, attachingidentifiers that connect this data set to the current case.

We now discuss issues specific to fetal scanning. A fetus more than nineweeks after conception moves, at intervals ranging from a few seconds toseveral minutes, depending on gestation stage and time of day(Development of daily rhythmicity in heart rate and locomotor activityin the human fetus: Kintraia, Zarnadze, Kintraia, and Kashakashvili, JCircadian Rhythms. 2005; 3: 5). In the ‘active’ time this is morefrequent (In ‘active’ hours, fetal locomotor activity augmented by 7-8times and was equal to 91 min and 10 sec, which corresponded to 16% ofthe recording time, loc. cit.). After a movement, unless the fetusreturns exactly to the previous position and orientation, a particularpoint of the fetal anatomy will create an echo from a different spatiallocation than before, and an echo from a particular point in space willrelate to a different point of anatomy. Anatomically guided search forparticular planar sections such as a section including a lengthwisecross-section of a femur is disrupted by such events, but onlymoderately so: usually the targeted body part has not moved or rotatedfar, so that a new search will not take long. However, integration intoa 3D image will have problems: indeed, no single 3D image cansimultaneously represent the ‘before’ and ‘after’ locations of the fetusand all its parts.

The fetus does move in a coherent (though not rigid) way, so that formost clinical purposes this problem can be handled, as shown in FIG. 12.We perform the steps in FIG. 11 from 1100 to 1131, then before thedisplay step 1240, corresponding precisely to step 1140 in FIG. 11, weinitialize 1200 a segment counter S to 0 and go to the incrementationstep 1201, which changes it to 1, and creates the first data segment.Proceeding from the step 1240, whether reached for the first time or ina later iteration of the loop, we acquire 1241 and test 1242 echo data,as in steps 1141 and 1142. However, we also test 1245 for fetallocomotion, optionally by the presence of motion blur inconsistent withthe known motion of the sensor, or by external electrocardiography (ECG)as performed by Kintraia et al. (loc. cit.), or by other such means asare known to or discovered by those skilled in the art. If there isevidence of locomotor activity since the previous enactment of step1241, we test 1247 whether locomotor activity is continuing. If it iscontinuing, we return directly to step 740, ignoring themovement-polluted data, acquire 1241 and test 1242 a new set ofultrasound data, and decide again 1245 whether locomotor activity iscontinuing. If locomotor activity is not continuing, we 1201 incrementthe counter S and create its data segment, and proceed again to 1240.

If at step 1245 there is no evidence of locomotor activity since theprevious enactment of step 1241, we proceed to step 1250, correspondingprecisely to step 1150 in FIG. 11. What follows corresponds to the stepsafter step 1150, thus 1151 and 1152 correspond now to 1251 and 1252,except that step 1153 is replaced by step 1253, adding the filled pointsto a record specific to the data segment numbered S. We then check 1260whether the regions 1 to S filled so far collectively cover the targetbox, with margin of overlap to allow for movement. If not, we proceed totrack the next (normally moved 1262) position of the hand held sensor,and repeat the display loop from the step 1240. If the regions 1 to S docover the target box, with a pre-set allowance of overlap, we go toencryption 1165 in FIG. 11 and proceed as in that figure. The creationof 3D image data is more complex than when scanning a static structure,as we discuss below in relation to step 1450 of FIG. 14.

Motion is also significant in scanning the heart, but there it is notmerely a complication, but an important object of study. With a fetusthe clinician primarily needs to know about anatomy (and blood flow,with Doppler sonography), which is well revealed by a static finalimage. The heart, however, beats: and the analysis of that beat can bediagnostically critical. For example, the ‘ejection fraction’ (EF) isthe volumetric fraction of blood pumped out of the left and rightventricle with each heartbeat or cardiac cycle. In the mathematicsallowed by medical imaging, EF is applied to either the right ventricle,which ejects blood via the pulmonary valve into the pulmonarycirculation, or the left ventricle, which ejects blood via the aorticvalve into the cerebral and systemic circulation. The preferred data setis four-dimensional, with an echo density at each point in a gridlabeled by (x,y,z,t), where (x,y,z) specifies a spatial location asusual, in a coordinate system where the patient is at rest, and t refersnot to absolute time but to the current phase of the heart. For example,if we take as reference the ‘R’ peak visible in an electrocardiogram(EKG), for a heart beating regularly at 72 beats per minute, the numbert refers to the time in minutes since the most recent R event, times 72.When t reaches 1, at the next R event, it resets to 0. If geometricallywe collect values in a hundred gridsteps in each of the x, y and zdirections, that gives one million points at which we need to estimatereflective properties from the echo data. For each ten timesteps intowhich we divide the period, ten million values are needed, and takelonger to collect. The record to be finally produced will includeestimates of values at times 0, T, 2T, etc., but these will often beinterpolated from echo data acquired at intermediate times.

In the static case, a small structure like the gall bladder (which wouldfit in an 8 cm×4 cm×4 cm box) can be sliced completely through by atypical fan-shaped acquisition zone, a sensor with 100 Hz frame ratecould move its acquisition region 212 once through in a second,collecting 100 planar slices and thus a full set of data. In a sweepthrough the heart, the zone 212 meets different points in the heartregion at different phases t (1)), and must pass through each again atother phases, from which we will interpolate data at phases 0, T, 2T,etc. FIG. 13 shows how we arrange this, in one embodiment of the presentinvention.

The steps 1300, 1302, 1303 and 1304 correspond exactly to the steps1100, 1102, 1103 and 1104, with the particularities that the anatomymodel A must include the surroundings of the heart in a generic patient(not the precise anatomy of the current patient), and the M fiducialpoints must suffice to localize it. (Points on the sternum, claviclesand scapulae are likely candidates, as are some rib points: but as somebones may be deeply covered in fatty tissue, it is desirable to allowsome flexibility of choice for the operator.) In step 1305 the operatorplaces fiducials at these M points, and also at least one electrode at apoint where a good EKG signal can be received. Optionally, some or allof the fiducial objects may also serve as electrodes.

In step 1307 the system analyses the signal from the one or moreelectrodes, and establishes that the heart is beating regularly, in thesense that successive R-R intervals are within a preset criterion ofmatching in length of time and shape of the EKG signal, so that phase tcan be coherently defined. (Disruptions such as cardiac arrhythmia arenot compatible with the reconstruction method described below.) In anembodiment it may be useful to include a non-linear matching ofsuccessive R-R intervals, to achieve a better match oft withphysiological phase than a simple ratio of times. From this point on, aphase label t can be attached to any record.

The initialization step 1310 corresponds precisely to 1110, thelocalization loop from 1320 to 1322 to 1323 to 1324 to 1320 is preciselythat of 1120 to 1122 to 1123 to 1124 to 1120. The fitting step 1330 isagain 1130, and the initialization 1331 is different only in setting upa four-dimensional box as ‘empty’. However, the display 1340 differsfrom 1120 in that it shows not a static setting but a dynamic one, withcues as to the phase t currently displayed. In a preferred embodiment,such cues are by movement or by change of color rather than by displayof numbers or a dial. In an exemplary manner of achieving this, thetarget box is shown as containing an open rhythmically movingthree-dimensional mesh (optionally but not necessarily resembling aheart), whose motion is phase locked to the current output of the EKG,and in which a node is colored green when at the current phase the pointit is in is ‘filled’, and otherwise colored red. Thus for example if theoperator first controls the sensor to move the acquisition region 812once downward through the heart, giving echo data corresponding tosuccessively lower planes in the physical space of the heart, thedisplay then shows a wave of green points moving down the mesh, whichremains mostly red. An efficient strategy for the operator is then tosweep downward again at the same speed, but with a different phase, suchas keeping the acquisition region 812 just above the green wave, or atsome chosen height above it. Some practice in a virtual training systemwill provide the operator with the necessary skill to completelyeliminate red points, in a minimal number of sweeps. Many other ways ofrepresenting information as to ‘filled’ and ‘un-filled’ points (x,y,z,t) will be evident to persons skilled in the art, within the spirit ofthe present invention. This technique extends straightforwardly toDoppler imaging (with three times the number of sweeps, from threesubstantially different sensor locations), with the merit that bloodflow data in a cardiac artery are acquired for a full cycle, without theoperator having to track the artery by moving the sensor to keep theartery in the acquisition zone 812. Furthermore, since a three-componentDoppler acquisition gives information about blood flow in everydirection, in the final four-dimensional data set the flow along such anartery can be followed along the whole length of that artery, not merelywhen it is toward or away from the sensor.

With the display 1340 as described above, the remainder of the logicalflow is identical to that of FIG. 11, replacing each initial ‘11’ in alabel by ‘13’, save that step 1352 stores the current phase t as well asthe current sensor location and orientation, step 1353 adds to the‘filled’ region of a four-dimensional box instead of to thethree-dimensional data structure of step 1152, and step 1360 checks forthe existence of unfilled parts of that four-dimensional box. Thesuccessive sweeps described above make it clear that the operator mustmake more sweeps, but this does not modify the flow organization of theacquisition system.

FIG. 14 shows how the local flow within an example of FIG. 11, 12 or 13is embedded in the distributed system. First, the clinical requirementsare acquired 1400 from a local EMR, or by local entry from a registereduser who is registered with the system as entitled to enter such dataand requests. (As an example, this might be a nurse at a one-doctorclinic, or the doctor in person. Such small clinical operations are ofparticular interest for the present invention, as they may not belicensed for ultrasound use by a government seeking to prevent sexdetermination, or may lack access to a sonographer trained with theanatomical skills required in using current equipment, particularly atlow cost.) The resulting data are passed 1410 to the local systemmanaging the sensor, which 1100 receives them and 1420 performs (withthe human operator) the steps in FIG. 11, 12 or 13, which 1166, 1266 or1366 return echo data, including the associated positions of the sensor.These are received 1430 by the distributed system, and transferred 1440to a processor with large computational power that performs thenumerically intensive step 1450 of creating a three-dimensional image,for the static and fetal cases, or a four-dimensional image, for thecardiac case. The latter is straightforward (though computationallyintensive), since each echo record in a particular direction from asensor in a particular location at a particular moment impliesreflectivity intensity values at that time (plus the sound travel time)at each point along the line in that direction from that position. Thefetal case is more complicated, since we have a number of ‘betweenlocomotion’ segments to deal with. In one embodiment the system firstchooses the largest such filled region as a reference, then finds forthe second largest the translation and rotation that achieves the bestmatching of reflectivity values where it overlaps with the first,discarding values that correspond to the wall or exterior of the womb.If there are more than two segments the system then finds for the thirdlargest the translation and rotation that achieves the best matching ofreflectivity values in the subset where it overlaps with the first andthe moved second, iterating until all segments have been included in afused three-dimensional layout. This is not sufficient in general,however, since often an overlap will correspond to a part of the fetusthat has changed non-rigidly, by bending at one or more joints, betweenthe data-segments involved. The system therefore fits a genericnumerical model of a fetal skeleton (preferably matched to gestationalage) to the most bone-like reflections in the fused layout, to identifywhich bone is which. With this information, and data on the anglesthrough which each joint can move, the system defines a mapping of thebones within the second segment to those of the first, and extends thisto the soft tissues to give a non-linear correspondence from one to theother that permits a fusion of the two three-dimensional partial images.(The art of such fusion of non-linearly related images is wellestablished, as is illustrated in 3D by ‘Image registration system andmethod’, US 20080095465 A1 to Mullick, Poston and Nagaraj, and manyother publications. In various contexts it is also known as montage,stitching and mosaic. Any means of achieving such a fusion of partialimages may be used within the spirit of the present invention.) If thereis a third data segment, the system similarly fuses the correspondingpartial image with the fusion of the first two, and so on until all datasegments have been used.

The choice of performing image creation at the stage 1450 is suggestedby the ‘cloud’ computing model, where a powerful processor does not sitidle when (for example) a sensor system to which the processor is boundis not performing a scan, and by the security advantage of the localnon-existence of images. This choice is thus our currently preferredembodiment of the invention, though the raw echo data are a largerdataset than the 3D or 4D image created from them. Limited bandwidth orhigh data transfer costs can thus alter the preferred solution, thoughthe problem is mitigated by the fact that the data need not betransferred in real time (as they would if, for example, a remoteoperator were to guide the sensor by controlling a local robot).Near-real-time response, aiming at a final clinician's report in minutesrather than a response in milliseconds, is less dependent on high anduninterrupted bandwidth.

The created three-dimensional or four-dimensional image, with theclinical record and patient ID (which optionally may exclude thepatient's real name or the scanning location, for security reasons) isdelivered 1460 to a station where a clinician has indicated availabilityfor immediate study and report. (The word ‘station’ here includes anysystem, from desktop to tablet or future-generation smart phone, that iscapable of receiving the data and supporting the necessary examination.)In a preferred embodiment the distributed system has access to a numberof clinicians and can seek an available one, greatly reducing thescheduling problems and random-arrival queue clusters that areinevitable with a single clinician, or a fixed allocation of particularclinicians to particular scanning sites.

The clinician studies 1470 the three-dimensional or four-dimensionalimage with tools well known to those skilled in the art, such as makingall but bone (or another tissue type) transparent, rotating the view,showing a slice at an arbitrary angle (unrelated to the orientation ofthe acquisition zone Z in any step 1141), quantifying distances orvolumes, etc. Automated or semi-automated tools of kinds well known tothe art, or additional innovations, may assist by quantifying distancessuch as femur length, volumes (including derived quantities such asejection fraction in the case of echocardiology), count and size ofgallstones or kidney stones, straightness and stage of healing for abroken bone, etc., as appropriate in the medical context. The clinicianthen prepares 1480 a report on the status of the patient, using theclinical record and the image data. In a preferred embodiment thisreport becomes part of a secure permanent record, so that anyinformation revealed remains subject to audit for clinical or regulatorypurposes. Two-dimensional or three-dimensional images may be included inthe report, in accordance with clinical utility and regulatoryconstraints. The distributed system then passes 1490 the report to thelocal operator, directly to the local clinician who requested the scan,or to the local electronic medical records system to be called up by anyauthorized person (in which case our preferred embodiment notifies thereport's arrival to those flagged as concerned with the particularcase).

It is important to distinguish here the various kinds of data that areintegrated in the present invention. We will refer to the following:

-   -   Pre-image data, which includes energy emitted by or reflected        from the tissue to be studied, such as light emitted by        fluorescence, or reflections sound or electromagnetic radiation        originating at the sensor, and further includes measured values        of such data, whether as an analog signal within the sensor such        as a voltage level, or as a digitized version of the signal        values, and further may include data concerning the emitted        signal, and reduced forms of such data. In the example displayed        in FIG. 7, pre-image data include the transmission timing data        721 of emitted pulses 707, the echoed sound 708, the values of        the analog signal 715 of the sensed echoed sound, the converted        digital signals 725 of these values, and the data flow 735        specific to particular emitted pulses 707 identified by the        comparison by the echo identifier 730, optionally including        Doppler comparison of the echo's frequency with that of the        emitted signal. Similarly, in frequency domain optical coherence        tomography (FD-OCT), pre-image data include the reference beam        travelling uninterrupted from an emitter, the beam reflected        from tissue, the interference signal created by physically        superposing these, the analog values created by sensing this        interference signal, their digitization and its Fourier        transform, and the identification of values corresponding to        particular phase differences of the two beams, and thus to        differences in time of travel.

The term ‘pre-image data’ includes time series of such data, and thetiming data that make it possible to integrate synchronous data ofdifferent types. Though many examples discussed in the presentdisclosure are specific to reflected ultrasound, nothing herein is to beconstrued as a restriction to that case.

-   -   Sensor-frame image data, constructed as in 740 from the        pre-image data by integrating transmission timing data 721 with        the echo data flow 735, to give reflection intensity values        along particular rays and thus (typically) pixel values in a        rectangular array that can be displayed in a planar image 106.        These planar pixel values may exist, but are not required to        exist, in an embodiment of the present invention.    -   Base-frame sensor position data, including but not limited to        (in various embodiments) analog values of electromagnetic wave        data, accelerometer data, images taken of or from the sensor,        and ultrasound signals, their digitization, and specifications        derived from such data of the location and orientation of the        sensor relative to a fixed base, which may be a signal emitter,        a camera, a cradle which holds the sensor in a reference        position, and other such bases as are well known to those        skilled in the art.    -   Fiducial position data, specifying the coordinates of the        fiducial objects in the base-frame coordinate system. These may        be obtained by their inclusion within the position-sensing        system (for example, a Polhemus ‘Flock of Birds’ can report the        positions of multiple objects relative to the same base, if each        contains a unit of the system), or by touching them with the        tracked image sensor. Optionally, one might sense their        locations using a separate subsystem, and transform the        resulting coordinates into a frame in which the image sensor's        position is also known, but our preferred embodiment gives them        in a common coordinate frame directly.    -   Patient-frame sensor position data, specifying the instantaneous        position of the sensor in a frame of reference in which the        fiducials have standardized locations, subject to anatomical        variation. For example, if for upper-body examination fiducials        are placed on the acromial extremities of a patient's clavicles,        at a distance D apart, a patient-oriented coordinate system        would typically require their locations to have the coordinates        (±D/2, Y, Z) with shared values of Y and Z. Similarly, it might        require a fiducial on the easily-located sixth vertebra to have        the location (0,0,0). These requirements would fix the        patient-frame coordinates of all other points. (Other such        schemes, some more complex, will be evident to persons skilled        in the art.) For the base-frame fiducial position data to have        this form would constrain the placement and orientation of the        base: to avoid this we prefer to position the base freely, and        determine from the sensed base-frame locations of the fiducials,        at anatomically specified positions, the affine transformation        between base-frame coordinates in general and the corresponding        patient-frame coordinates.

Note that since this transformation depends on parameters such as D, theexpected region of interest 505 occupied by a target tissue cannot begiven as (for example) a polyhedron with a fixed set of coordinates forits corners. For instance, in a typical adult the apex of the heart lies8 cm to 9 cm from the mid-sternal line, but in an infant both D and thisdistance are smaller. The computation of the target region of interest505 in patient frame coordinates is thus a substantial step of theinvention, requiring application of both mathematical and anatomicalknowledge: however, with the task once specified, the details of thealgorithm can be developed by any person or group sufficiently skilledin the mathematical and anatomical arts.

In the terminology used here, this ‘target region of interest’ 505, withrespect to which data are to be acquired, is distinct from the‘acquisition region’ 550 with respect to which at a specific moment thesensor is acquiring data.

-   -   Patient-frame image data, giving pixel values in a        three-dimensional array specified with reference to the frame of        reference in which the fiducials have their required values.        -   These data may be assembled from sensor-frame image data, as            created 740 in FIG. 7, or directly from pre-image data and            sensor position data as defined above.    -   Display-frame sensor data, derived from the base-frame sensor        position data, which determine where in the display exemplified        in FIG. 5 the acquisition region 550, in terms of the graphical        coordinate system used in the display. (Typically, these        coordinates specify lateral and vertical position relative to        the screen, and ‘depth’ or distance from the viewer.) In our        preferred embodiment the transformation from base frame to        display frame arranges that displayed movements appear to the        user to be parallel to the sensor movements that the user        directly wills, sees and feels: the determination of this        transformation can be done once for all, in calibration of the        system, if the base and display are physically fixed in a        particular geometric relationship.

These distinctions are important both for the effective design of anembodiment of the present invention, and for the aspect of imagesecurity. Until the stage at which sensor-frame or patient image dataexist, there exists nothing that can be transferred to a display andinterpreted by a human as information about a fetus, a heart, or amalignant growth. Intercepting the pre-image data alone does not sufficefor imaging: it must be integrated with at least the-information of thetransmission timing data 721 for a planar image, or additionally withsensor position data 719 for a 3D image (unless a more costly 3D sensoris used), before any breach of anatomical security can occur. Creatingan interception system capable of performing such integration wouldinvolve a similar effort to the creation of an entirely independentimaging system, and its sale and use would be restricted by the sameIndian laws that already govern local-display ultrasound system.

This is in great contrast to such imaging devices as an indirectophthalmoscope, which illuminates the inner eye, collects light andphysically forms an image, which may immediately be viewed by theclinician or captured by a still or video camera. The level ofprocessing required before in any sense an image exists provides amultiplicity of options for security, which will be evident to thoseskilled in the art, by encryption or physical inaccessibility at anypoint in the pre-image information flow. Encryption or inaccessibilityeven of low-bandwidth parts of the pre-image data, such as the timing orpositional data, is sufficient to block unauthorized image creation.

A computer is a programmable machine, It. need not be a singleprogrammable component, hut may be multiple programmable components incommunication with each other as a unit still defined as a computer.Tins means that the computer can execute a program, code or programmedlist. of instructions/commands and respond to new instructions that itis given When referring to a desktop model, the term “computer”technically only refers to the computer itself not the monitor,keyboard, mouse and other ancillary addenda. Still, it is acceptable torefer to everything together as the computer. Some of the major parts ofa personal computer (or PC) include the motherboard, CPU, memory (orRAM), flash drive, EPROM, hard drive, audio card, speakers, display andvideo card or graphics processing unit (GPU). In the present disclosure,“the computer” refers to all such components, including processors thatmay be located in the hand-held sensor itself (in particular, though notexclusively, to perform local encryption), as well as those in thelarger system that manage display, distant communication, and the otherfunctions common to current lap-top and desktop systems. While personalcomputers are by far the most common type of computers today, there areseveral other types of computers, and we do not intend the term to referto only one type.

A general apparatus useful in practices of the present technologyincludes:

an on-site non-invasive scanning device for producing image datarelating to an internal target volume of a subject;

a computer configured to receive the image data from the non-invasiveimaging device and execute code to track motion of the non-invasivescanning device (e.g., by executing code, processing data, analyzingdata, constructing useful information as text or image, etc.);

a display screen in communication with the computer configured toreceive image data (or information in any useful form) from the computerand provide image (including text) feedback to an operator of thenon-invasive imaging device that comprises information associated with adisplay of motion and imaged content through a three-dimensional virtualmodel of an image-acquisition region of the non-invasive scanning devicethrough a three-dimensional virtual model of the internal target volume.The computer is configured to acquire data from the non-invasive imagingdevice as the operator moves the non-invasive scanning device, and thereis an information transmission link for transmitting the acquired datato at least one computer system that is not on-site. The at least onecomputer system that is not on-site (this may still be a relativelylocal computer, in the same town, village, city or country as theon-site non-invasive imaging system) being a geographically andphysically separated computer system configured to receive, process andanalyze the transmitted data; and

the at least one geographically separated computer system providingprocessed and/or analyzed data to an on-site recipient computer.

A desirable feature of the data transformations above, for the purposesof the present invention, is that they occur in ‘real time’, which wemay define variously for different subsystems (instantaneity beingphysically impossible). For position data we define it here as involvinga delay of no more than 100 milliseconds between the moment that thesensor is at a particular position, and the moment when thecorresponding representation 550 of the acquisition zone appears to theuser. (A preferred embodiment would limit the delay to around 20milliseconds, so that the display is at most one image-refresh behindthe actual position.) Anatomically guided ultrasonography requires asimilar limit on the delay, so that the planar image 106 follows thechanges in sensor position without perceptible delay, which would makethe user's task more difficult and slow. The present invention can beslightly more relaxed in this respect, since the image data are notdisplayed to the user: however, where we display derived image data(such as bone locations, or the modifications illustrated in FIGS. 9 and10), the additional processing makes the overall condition of ‘noperceptible delay’ more stringent. Even where we do not displayimage-derived data, we provide feedback about data quality, which theuser should be able to correct within seconds.

We therefore desire that each of the steps in the above processing frompre-image data to image data, as well as from base-frame sensor positiondata to displayed positions of the acquisition region, determination ofdisplayable regions where further acquisition is needed, and optionalextraction of image features such as bones. Many of these steps could beperformed by a human or a team of humans, manipulating symbols on paper.A skilled algebraist, given the list of sensed fiducial positions andthe corresponding anatomical locations with their adjustablecoordinates, could extract in less than an hour the coefficients of theaffine transformation A between base-frame sensor position data andpatient-frame sensor position data, and apply A to transform in under aminute a new set of (x,y,z) and (roll, pitch, yaw) base-frame data intopatient-frame data. However, this would be of no value for the presentinvention. Although the mathematical procedures involved predate digitalcomputers, they combine with a computer to something new, an embeddedalgorithm producing results in real time. In some cases the algorithmsused here are known to those skilled in the art: in particular, thedisplay of a virtual object moving in synchrony with a real object is inwide use, from the 2D cursor moving with a mouse to 3D objects in videogames. We do not claim novelty for all the real-time embedded algorithmsused within all steps of the present invention, but in no case do theyperform on a computer what was previously done for the same purposewithout one. The real-time speed permitted by a computer allows theirassembly into a system which could not be embodied without one, so thattheir novelty must be assessed relative to other computer-basedprocedures: corresponding manual procedures, achieving the samereal-time result, simply do not exist. An embedded algorithm may consistof a set of specialized fixed connections between logical elements of acomputer chip such as a digital signal processor (DSP) or anapplication-specific integrated circuit (ASIC), as code stored in memorythat instructs the operating system of a computer to perform specificlogical and numerical steps, or hybrid elements such as afield-programmable gate array (FPGA).

All of these fall within the meaning used here of ‘real-time embeddedalgorithm’, which we define as ‘exercised’ when the computer orcomponent performs the steps required.

Although specific examples of components, apparatus, materials,frequencies and times are provided, one skilled in the art is aware thatvariations from those specific examples may be used in the practice ofthe generic invention described and enabled herein.

What is claimed is:
 1. A method of acquiring medical images, the methodcomprising: (i) an initialization step of selecting a target tissue froma menu of such tissues; (ii) a computer system recalling from storagespecification of anatomy around the selected target tissue, and a listof anatomically defined fiducial positions required for use with theselected target tissue. (iii) an on-site user operating a non-invasiveimaging device on a subject to produce pre-image data relating to aninternal region of the subject and providing that pre-image data to thecomputer system; (iv) repetitively sensing and reporting location andorientation of the non-invasive imaging device in a coordinate frame setby a fixed base of a sensing system; (v) placing at least three externalfiducials on the subject at locations in a predetermined relationship tobody part structure within the internal region of the patient; (vi)registering as received fiducial data positions of the fiducials on thesubject, with respect to a base frame in which position of thenon-invasive imaging device is repetitively obtained; (vii) the computersystem exercising a real-time algorithm on received fiducial data toestablish a transformation between coordinates relative to the baseframe and anatomy-model coordinates in terms of which a model of anatomysurrounding the target tissue has been defined; (viii) the computersystem determining, within the anatomy-model coordinates, a geometrictarget region containing the selected target tissue; (ix) the computersystem repetitively determining the location and orientation of thenon-invasive imaging device with respect to the base frame; (x) thecomputer system repetitively transforming the location and orientationof the non-invasive imaging device to the anatomy-model coordinates;(xi) the computer system recording and integrating the pre-image data,and sensor location and orientation data within a target region overwhich an acquisition region of the non-invasive imaging device is moved;(xii) the computer system displaying to the on-site user on a visualdisplay system a geometric representation of the geometric targetregion; (xiii) the computer displaying to the on-site user on the videodisplay system a geometric representation of sub-regions within theinternal region of the subject, the sub-regions indicating a currentlocation and orientation of an acquisition region for which thenon-invasive imaging device is capable of acquiring pre-image data andfrom which the computer exercises a real-time algorithm to create animage; (xiv) the non-invasive imaging device collecting pre-image datafrom a series of acquisition regions within the internal region of thesubject; (xv) the computer exercising a real-time algorithm to pro-videa created image of the target region of the internal volume; (xvi) thecomputer exercising at least one real-time algorithm to apply a localimage quality test of whether the data in a plurality of regions in thecreated image satisfy predetermined numerical criteria of quality;(xvii) the computer creating a geometric representation of points in thetarget region of the subject which have failed at least one predefinedimage quality test; (xviii) the system displaying the geometricrepresentation to the on-site user; and (xix) the on-site user beinginvited to acquire further pre-image data, which are processed as inearlier steps, and the resulting image data added to the created image.2. The method of claim 1, where the subject is a training dummy havingan anatomically correct region of a human body and where the trainingdummy is empty and only fiducial position data, base-frame sensorposition data, fiducial position data, patient-frame sensor positiondata, and display-frame sensor data are collected or constructed, andthe user receives feed-back derived from the collected or constructeddata and from computed values of local image quality based on thecollected or constructed data and a model of a per-formed acquisitionprocess, on whether points in the geometric target region have beenswept within the acquisition region of the device in a manner whichpasses local image quality tests.
 3. A method of training a user in theuse of a system for obtaining medical images comprising: an on-siteoperator manually operating a non-invasive scanning device to produceimage data relating to an internal geometric target region of a subject;tracking the motion of the non-invasive scanning device; guiding theoperator by feedback that comprises motion of a three-dimensionalvirtual model of the acquisition region of the non-invasive scanningdevice through a three-dimensional virtual model of the geometric targetregion; and reporting aspects of the motion that would reduce thequality of the data that such motion would acquire.
 4. The method ofclaim 3, wherein the acquired data are secured against local inspectionas an image.
 5. An apparatus for performing the method of claim 1 ofacquiring medical images and evaluating performance characteristics inobtaining the medical images comprising: an on-site non-invasivescanning device for producing image data relating to an internal targetvolume of a subject; a computer configured to receive image data fromthe non-invasive imaging device and execute code to track motion of thenon-invasive scanning device; a display screen in communication with thecomputer configured to receive image data from the computer and provideimage feedback to an operator of the non-invasive imaging device thatcomprises a display of motion through a three-dimensional virtual modelof an image-acquisition region of the non-invasive scanning devicethrough a three-dimensional virtual model of the internal target volume;the computer configured to acquire data from the non-invasive imagingdevice as the operator moves the non-invasive scanning device; aninformation transmission link for transmitting the acquired data to atleast one computer system that is not on-site; the at least one computersystem that is not on-site being a geographically separated computersystem configured to receive, process and analyze the transmitted data;and the at least one geographically separated computer system providingprocessed and analyzed data to an on-site recipient computer.